Data Engineer

Job description

Roles and Responsibilities • Design, develop, and maintain large-scale data pipelines using Azure Databricks, Python, and Data Factory. • Collaborate with cross-functional teams to gather requirements and deliver high-quality solutions on time. • Develop complex SQL queries to extract insights from massive datasets stored in Azure Blob Storage. • Implement data quality checks and validation processes to ensure accuracy and integrity of the data. • Troubleshoot issues related to pipeline failures or errors in real-time. Desired Candidate Profile • 9-14 years of experience as a Data Engineer with expertise in Azure Databricks, Python, and Data Factory. • Strong understanding of big data technologies such as Hadoop Distributed File System (HDFS), Apache Spark, etc. . • Experience working with cloud-based platforms like AWS or GCP is an added advantage. • Proficiency in writing complex SQL queries for querying large datasets stored in Azure Blob Storage.