Data Engineer - Azure Databricks & Data Factory

Job description

**Key Responsibilities** - Design and develop **end-to-end data pipelines** using **Azure Data Factory and Azure Databricks** - Build and maintain **data transformation workflows using PySpark** - Ensure **data quality, reliability, and performance** of pipelines in production environments - Work with **large structured and unstructured datasets** - Collaborate with **platform engineers, architects, and BI teams** to deliver reusable data assets - Optimize data pipelines for **performance, scalability, and cost efficiency** - Implement data ingestion and transformation logic across multiple sources - Participate in **release cycles, deployments, and monitoring of data pipelines** **MustHave Skills** **Data Engineering** - Strong experience in: - **ETL/ELT pipeline development** - Data ingestion and transformation - Experience working with **large-scale data systems** **Azure Data Platform** - Hands-on experience with: - **Azure Data Factory (ADF)** - **Azure Databricks (PySpark)** **Programming** - Strong expertise in: - **Python** - **SQL** **Big Data Processing** - Experience with: - **PySpark / Apache Spark** - Strong understanding of distributed data processing **DevOps & Tools** - Familiarity with: - **CI/CD pipelines** - **Git / version control** **GoodtoHave Skills** - Experience with: - **Data platform architecture patterns** - Exposure to: - Cloud optimization and performance tuning - Experience working in: - Agile environments