Databricks Data Engineer

Job description

ROLE OVERVIEW We are seeking a highly skilled Databricks Engineer to design, build, and optimize scalable data pipelines KEY RESPONSIBILITIES • Design, develop, and maintain end-to-end data pipelines using Databricks and PySpark. • Build Lakehouse architecture using bronze, silver, and gold layers. • Implement Delta Lake with ACID transactions and schema enforcement. • Develop and manage Delta Live Tables (DLT) pipelines. • Implement Auto Loader for incremental ingestion. • Build streaming pipelines using Structured Streaming. • Orchestrate jobs using Databricks Workflows. • Apply Lakehouse data modeling for analytics. • Enforce governance using Unity Catalog. • Optimize SQL queries and Spark jobs. PRIMARY SKILLS (MUST-HAVE) • Databricks (Delta Lake, Delta Live Tables, Unity Catalog, Workflows) • PySpark and Apache Spark • Structured Streaming • Auto Loader • SQL • Lakehouse Data Modeling GOOD-TO-HAVE SKILLS • Azure: ADF, Synapse, Purview, Fabric • AWS: Glue, Lambda, Step Functions • Airflow • DBT, Five Tran, Informatica • Kafka • Power BI • Collibra, Alation • GCP Big Query QUALIFICATIONS • Bachelor’s or master’s degree in computer science, Data Engineering, or related field. • Proven experience building cloud-based data pipelines. • Strong analytical and problem-solving skills. • Excellent communication and collaboration skills.