Azure Databricks Architect

Job description

Role: Azure Databricks Architect Exp Level: 8-10 Years Work Location: Bangalore We are seeking a Databricks Architect to design and lead our enterprise data and AI platform built on Databricks. You will define the architecture for large-scale data engineering, analytics, and machine learning workloads on the Lakehouse platform. Key Responsibilities • Design end-to-end Lakehouse architectures on Databricks, including ingestion, transformation, storage, and consumption layers. • Define best practices for Delta Lake, Unity Catalog, and workspace governance across environments. • Architect scalable data pipelines using Spark, Delta Live Tables, and Databricks Workflows. • Establish security, access control, and data governance standards within Databricks and the underlying cloud platform (Azure/AWS/GCP). • Guide performance tuning and cost optimization of Spark jobs and cluster configurations. • Collaborate with data engineering, data science, and ML teams to enable a unified data and AI platform. • Define CI/CD and DevOps practices for Databricks notebooks, jobs, and infrastructure (Repos, Terraform). • Provide architectural oversight and mentorship to data engineering teams. • Partner with stakeholders to align the Databricks platform roadmap with business and analytics needs. Required Skills & Qualifications • 8-10 years of overall data engineering/architecture experience, with at least 4.5 years hands-on with Databricks. • Deep expertise in Apache Spark, Delta Lake, and the Databricks Lakehouse architecture. • Strong experience with Unity Catalog, workspace administration, and cluster/job management. • Hands-on experience with at least one hyperscaler (Azure, AWS, or GCP) underlying the Databricks deployment. • Proficiency in Python and/or Scala, and strong SQL skills. • Experience designing batch and streaming data pipelines at enterprise scale. • Solid understanding of data governance, lineage, and security practices in a Lakehouse environment. • Strong communication skills and experience leading technical teams. Preferred Qualifications (Good to Have) • Databricks Certified Data Engineer Professional or Databricks Certified Solutions Architect. • Experience with MLflow and enabling ML/AI workloads on Databricks. • Exposure to orchestration tools such as Airflow or Azure Data Factory. • Experience migrating legacy data warehouses to a Lakehouse architecture.