Lead Engineer

Job description

Job Title: Data Engineering Lead – Role Summary: We are looking for a Data Engineering Lead to lead a team of data engineers in building and delivering scalable ingestion and transformation pipelines on Databricks + AWS , ensuring engineering execution aligns with the platform's architecture and delivery timelines. Key Responsibilities • Lead a team of Data Engineers delivering Bronze/Silver/Gold pipelines across batch, near-real-time, streaming, and CDC patterns • Drive hands-on implementation and code/design reviews for ingestion pipelines ( AWS Glue, DMS, DataSync ) and streaming jobs ( Kinesis, Kafka, or Databricks Structured Streaming ) • Own sprint planning, task allocation, and delivery tracking for the engineering pod • Ensure adherence to Unity Catalog governance standards, data quality checks, and orchestration best practices ( Airflow/MWAA ) • Troubleshoot complex pipeline/performance issues and mentor junior engineers • Collaborate closely with the Data Architect to translate architecture into engineering execution • Support client/stakeholder updates on engineering progress and technical blockers Must-Have Skills • 7–10 years in data engineering, including 2+ years leading a team/pod • Strong hands-on Databricks/Spark , Delta Lake , SQL , and Python/PySpark expertise • Strong AWS data services experience: Glue, S3, DMS; working knowledge of Kinesis/MSK or Kafka • Solid understanding of medallion architecture, CDC ingestion, and orchestration tools • Proven ability to manage delivery timelines and mentor a team Good-to-Have: Exposure to vector databases, RAG pipeline data prep, or agentic/AI tooling (Bedrock, Mosaic AI).