Databricks

Job description

⃣ Sr. Data Engineers – Databricks Pipeline Developer Role Overview Build and maintain data pipelines in Databricks for ingestion, cleansing, enrichment, and storage. Key Responsibilities • Develop PySpark pipelines for • API ingestion • FTP file ingestion • Data cleansing • Enrichment transformations • Implement incremental & batch loads • Develop Bronze → Silver → Gold transformations • Translate legacy SQL stored procedures into Spark jobs • Optimize Delta Lake tables • Implement validation checks • Support testing & reconciliation Required Skills • Strong PySpark / Spark SQL • Strong SQL (read & interpret legacy procedures) • Experience with API integration • Experience handling flat files (FTP ingestion) • Experience with Delta Lake • Knowledge of partitioning & optimization