Azure Databricks Engineer

Job description

Skill: Azure Databricks Engineer Location: Pune and Bangalore Experience: 8 to 12 years Job Description: Deliver and operate scalable reliable data engineering solutions on Databricks to support analytics AI and downstream consumption Key Responsibilities • Design and build ETL/ELT pipelines using Databricks, PySpark and SQL • Implement batch and incremental data processing patterns for enterprise datasets • Apply data engineering best practices for performance scalability and reliability • Develop and optimize Spark jobs notebooks and workflows in Databricks • Ensure data quality validation and reconciliation across pipelines • Support data modeling and transformation for analytics and reporting use cases • Integrate Databricks pipelines with upstream and downstream systems • Participate in production support monitoring and incident resolution Core Skills Required • Strong handson experience with Databricks and Apache PySpark • Programming skills in Python, PySpark and SQL for efficient data transformation and processing ETL • Solid understanding of ETL/ELT patterns and data pipeline orchestration • Big Data Processing Experience with big data frameworks specifically Apache Spark for processing large datasets • Data Warehousing Modelling Understanding of data warehousing principles and scalable data modelling techniques • Experience with data engineering best practices and optimization techniques • Handson exposure to enterprise data engineering workflows • Experience supporting production data pipelines at scale • Familiarity with analytics reporting or AIdriven data consumption use cases Good to have • Experience in working with Opensource storage formats such as Apache Delta Apache Parquet would be beneficial • Awareness of compliance regulations relevant to data handling and processing • Knowledge of data security practices including encryption access control and data governance • Knowledge of other Azure services that integrate with ADF and ADB such as Azure Functions Azure Logic Apps and Azure Event Hubs