Azure Data Engineer (5 To 10 years)- Chennai/ Bangalore

Job description

Job Requirements\\* • Azure data factory, Databricks, synapse, delta lake development with hands-on coding experience • Implement ETL solution to integrate, transform and load data from various sources into data lake and data warehouse • Hands-on expertise in python, pyspark and sql for large scale data processing • Optimize and tune data pipelines for performance and scalability • Ability to write complex SQL queries • Collaborate with business analysts and business stakeholders to gather requirements and ensure data quality and availability. • Good understanding of Agile Methodologies and DevOps Culture • Strong Problem-solving skills . Key Responsibilities\\* • Design, develop, and deploy scalable data pipelines using Databricks (PySpark, Spark SQL), Azure Synapse, Azure Data Factory, and other Azure data services. • Implement ETL/ELT processes to ingest, transform, and load data from various sources into data lakes and data warehouses. • Optimize and tune data pipelines for performance and scalability. • Write and optimize complex SQL queries for data extraction, transformation, and analysis. • Use PySpark for large-scale data processing and analytics. • Implement data partitioning, bucketing, z-ordering, liquid clustering and indexing strategies for efficient data retrieval. • Integrate data from multiple sources, including structured, semi-structured, and unstructured data. • Work with APIs, streaming data, and batch processing to ensure seamless data integration. • Implement data governance practices to ensure data quality, consistency, and security. • Monitor and troubleshoot data pipelines to ensure data accuracy and availability. • Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions. • Work closely with DevOps teams to deploy and monitor data pipelines in production environments. • Document data pipelines, workflows, and processes for knowledge sharing and future reference. • Maintain up-to-date documentation on data architecture and data models.