Data Engineer

Job description

**Role & responsibilities** Roles & Responsibilities - Design, develop, and maintain scalable ETL/ELT pipelines using **Python**, **PySpark**, and **Azure Databricks**. - Build and orchestrate end-to-end data workflows using **Azure Data Factory (ADF)**. - Develop, optimize, and maintain SQL queries, stored procedures, and data models to support business and analytical requirements. - Ingest, transform, and process large volumes of structured and unstructured data from multiple sources. - Develop, test, troubleshoot, and maintain **REST APIs** to enable seamless data integration across applications and platforms. - Build and deploy **Azure Functions** to support serverless data processing and event-driven workflows. - Collaborate with data architects, business analysts, and application teams to understand data requirements and deliver scalable solutions. - Monitor, troubleshoot, and optimize data pipelines for performance, reliability, and scalability. - Implement data quality, validation, and error-handling mechanisms to ensure data accuracy and integrity. - Follow best practices for cloud-native development, version control, documentation, and CI/CD in the Azure ecosystem. - Participate in Agile ceremonies, code reviews, and technical discussions to ensure high-quality deliverables. **Preferred candidate profile** Preferred Candidate Profile - **47 years** of experience as a **Data Engineer** with hands-on experience in Azure-based data platforms. - Strong proficiency in **Python**, **PySpark**, and **SQL** for large-scale data processing. - Proven experience with **Azure Databricks** and **Azure Data Factory (ADF)** for building enterprise-grade data pipelines. - Good understanding of Azure cloud services and cloud-based data engineering best practices. - Experience developing and maintaining **REST APIs** and integrating them into enterprise applications. - Hands-on experience developing and deploying **Azure Functions** in serverless architectures. - Strong knowledge of data modeling, database design, query optimization, and performance tuning. - Experience working with Git, CI/CD pipelines, and Agile/Scrum development methodologies. - Excellent analytical, debugging, and problem-solving skills with the ability to troubleshoot complex data issues. - Strong communication and collaboration skills, with experience working in cross-functional and globally distributed teams.