Azure Databricks Engineer

Job description

**Skill: Azure Databricks Engineer** **Location: Pune and Bangalore** **Experience: 8 to 12 years** **Job Description:** Deliver and operate scalable reliable data engineering solutions on Databricks to support analytics AI and downstream consumption **Key Responsibilities** - Design and build ETL/ELT pipelines using Databricks, PySpark and SQL - Implement batch and incremental data processing patterns for enterprise datasets - Apply data engineering best practices for performance scalability and reliability - Develop and optimize Spark jobs notebooks and workflows in Databricks - Ensure data quality validation and reconciliation across pipelines - Support data modeling and transformation for analytics and reporting use cases - Integrate Databricks pipelines with upstream and downstream systems - Participate in production support monitoring and incident resolution **Core Skills Required** - Strong handson experience with Databricks and Apache PySpark - Programming skills in Python, PySpark and SQL for efficient data transformation and processing ETL - Solid understanding of ETL/ELT patterns and data pipeline orchestration - Big Data Processing Experience with big data frameworks specifically Apache Spark for processing large datasets - Data Warehousing Modelling Understanding of data warehousing principles and scalable data modelling techniques - Experience with data engineering best practices and optimization techniques - Handson exposure to enterprise data engineering workflows - Experience supporting production data pipelines at scale - Familiarity with analytics reporting or AIdriven data consumption use cases **Good to have** - Experience in working with Opensource storage formats such as Apache Delta Apache Parquet would be beneficial - Awareness of compliance regulations relevant to data handling and processing - Knowledge of data security practices including encryption access control and data governance - Knowledge of other Azure services that integrate with ADF and ADB such as Azure Functions Azure Logic Apps and Azure Event Hubs