SENIOR DATA ENGINEER - Python

Job description

**Title** Senior Data Engineer **Skills (must have)** - 5+ years of professional experience in data engineering or related fields. - Strong programming skills in Python, SQL and PySpark. - Advanced experience building & optimizing ETL/ELT pipelines using Azure and open-source tools: - Azure Data Factory (ADF) - Azure Databricks (Spark, Delta Lake) - Apache Airflow - Azure Functions or Azure Synapse Pipelines - Expert-level SQL development, including complex queries, stored procedures, analytical functions and performance tuning. - Strong experience with Azure Snowflake, including: - Warehouse tuning - Cost/performance optimization - Snowpipe, Streams, Tasks - Snowpark for advanced processing - Experience building scalable data models (Kimball, Data Vault, Lakehouse). - Strong experience with Azure Cloud ecosystem, including: - Azure Data Lake Storage (ADLS Gen2) - Azure Synapse Analytics (Serverless & Dedicated SQL Pools) - Azure Databricks - Azure Key Vault - Azure Event Hub / IoT Hub - Azure Monitor / Log Analytics - Experience integrating APIs, streaming data and external data sources. - Strong understanding of data governance & data quality frameworks: - Microsoft Purview (cataloging, lineage, classifications) - Collibra - Experience with testing frameworks: unit tests, integration tests, Great Expectations, dbt tests. - Hands-on with CI/CD pipelines using: - GitHub Actions - Azure DevOps - Jenkins - Experience working with Infrastructure as Code tools: - Terraform or Bicep **Skills (good to have)** - Experience with Azure Kubernetes Service (AKS) or Dockerized workloads. - Experience building high-performance APIs using FastAPI, Flask, or Django. - Experience with real-time/streaming systems: - Azure Event Hub - Azure Stream Analytics - Kafka - Familiarity with Microsoft Fabric including: - Lakehouse - Data Pipelines - Warehouses - Experience with ML Ops or Feature Store integration (Databricks Feature Store, Azure ML). - Experience with security frameworks (RBAC, ABAC, managed identities). Responsibilities - Lead the design, development and deployment of high-performance ETL/ELT pipelines on Azure and Snowflake. - Partner with business stakeholders, architects and data consumers to understand requirements and build scalable data solutions. - Design and optimize Azure Lakehouse solutions using: - ADLS Gen2 - Delta Lake - Azure Databricks - Synapse Analytics - OneLake (if using Microsoft Fabric) - Design and optimize data models to support BI, analytics and machine learning. - Build and maintain data ingestion frameworks for: - Batch pipelines - Real-time pipelines - API-based ingestion - Implement robust data quality, data validation and metadata management processes. - Drive performance tuning across Spark jobs, Snowflake warehouses, SQL pool queries and cost optimization. - Monitor and maintain production data systems using Azure-native monitoring tools. - Define, enforce and improve engineering standards, including automation, CI/CD and IaC best practices. - Collaborate closely with DevOps/Platform teams to automate data infrastructure deployments. - Troubleshoot complex issues across distributed systems, cloud networks and data platforms. - Mentor junior and mid-level engineers, conduct code reviews and guide best practices. - Maintain clear architectural and technical documentation. Senior-Level Behavioral Expectations - Provide technical leadership and drive decision-making during architecture and design discussions. - Effectively communicate with both technical and non-technical audiences. - Strong ownership mindset and proactive approach to solving technical challenges. - Ability to break down complex requirements into actionable engineering tasks. - Promote continuous learning, experimentation and innovation within the data engineering team.