SENIOR DATA ENGINEER - Python

Job description

Title Senior Data Engineer Skills (must have) • 5+ years of professional experience in data engineering or related fields. • Strong programming skills in Python, SQL and PySpark. • Advanced experience building & optimizing ETL/ELT pipelines using Azure and open-source tools: • Azure Data Factory (ADF) • Azure Databricks (Spark, Delta Lake) • Apache Airflow • Azure Functions or Azure Synapse Pipelines • Expert-level SQL development, including complex queries, stored procedures, analytical functions and performance tuning. • Strong experience with Azure Snowflake, including: • Warehouse tuning • Cost/performance optimization • Snowpipe, Streams, Tasks • Snowpark for advanced processing • Experience building scalable data models (Kimball, Data Vault, Lakehouse). • Strong experience with Azure Cloud ecosystem, including: • Azure Data Lake Storage (ADLS Gen2) • Azure Synapse Analytics (Serverless & Dedicated SQL Pools) • Azure Databricks • Azure Key Vault • Azure Event Hub / IoT Hub • Azure Monitor / Log Analytics • Experience integrating APIs, streaming data and external data sources. • Strong understanding of data governance & data quality frameworks: • Microsoft Purview (cataloging, lineage, classifications) • Collibra • Experience with testing frameworks: unit tests, integration tests, Great Expectations, dbt tests. • Hands-on with CI/CD pipelines using: • GitHub Actions • Azure DevOps • Jenkins • Experience working with Infrastructure as Code tools: • Terraform or Bicep Skills (good to have) • Experience with Azure Kubernetes Service (AKS) or Dockerized workloads. • Experience building high-performance APIs using FastAPI, Flask, or Django. • Experience with real-time/streaming systems: • Azure Event Hub • Azure Stream Analytics • Kafka • Familiarity with Microsoft Fabric including: • Lakehouse • Data Pipelines • Warehouses • Experience with ML Ops or Feature Store integration (Databricks Feature Store, Azure ML). • Experience with security frameworks (RBAC, ABAC, managed identities). Responsibilities • Lead the design, development and deployment of high-performance ETL/ELT pipelines on Azure and Snowflake. • Partner with business stakeholders, architects and data consumers to understand requirements and build scalable data solutions. • Design and optimize Azure Lakehouse solutions using: • ADLS Gen2 • Delta Lake • Azure Databricks • Synapse Analytics • OneLake (if using Microsoft Fabric) • Design and optimize data models to support BI, analytics and machine learning. • Build and maintain data ingestion frameworks for: • Batch pipelines • Real-time pipelines • API-based ingestion • Implement robust data quality, data validation and metadata management processes. • Drive performance tuning across Spark jobs, Snowflake warehouses, SQL pool queries and cost optimization. • Monitor and maintain production data systems using Azure-native monitoring tools. • Define, enforce and improve engineering standards, including automation, CI/CD and IaC best practices. • Collaborate closely with DevOps/Platform teams to automate data infrastructure deployments. • Troubleshoot complex issues across distributed systems, cloud networks and data platforms. • Mentor junior and mid-level engineers, conduct code reviews and guide best practices. • Maintain clear architectural and technical documentation. Senior-Level Behavioral Expectations • Provide technical leadership and drive decision-making during architecture and design discussions. • Effectively communicate with both technical and non-technical audiences. • Strong ownership mindset and proactive approach to solving technical challenges. • Ability to break down complex requirements into actionable engineering tasks. • Promote continuous learning, experimentation and innovation within the data engineering team.