Walk-in || Senior Data Engineer - Databricks

Job description

EXPERIENCE - 6+ YEARS **Key Responsibilities** Data Pipeline Development - Design, develop, and optimize scalable batch and streaming data pipelines using Databricks. - Build robust ETL/ELT frameworks using PySpark and Spark SQL. - Develop ingestion pipelines from: - APIs - Kafka - Cloud Storage - Databases - Event-driven sources Lakehouse Architecture - Design and maintain Medallion Architecture: - Bronze Layer - Silver Layer - Gold Layer - Implement Delta Lake features including: - ACID Transactions - Schema Evolution - MERGE - UPSERT - Time Travel Data Engineering & Modeling - Build dimensional and Lakehouse data models. - Support analytical and reporting workloads. - Ensure data quality, reconciliation, lineage, and governance. Performance Optimization - Optimize Spark workloads using: - Partitioning - Caching - Cluster Tuning - Query Optimization - Improve scalability and reduce processing costs. Governance & Security - Implement governance practices using: - Unity Catalog - Metadata Management - Access Controls - Compliance Standards Orchestration & Automation - Integrate pipelines using: - Airflow - Cloud Composer - Azure Data Factory - Support CI/CD and deployment automation. **Must-Have Skills** Databricks - Strong hands-on experience with: - Databricks - Databricks Workflows - Cluster Management - Performance Optimization Big Data - Expertise in: - Apache Spark - PySpark - Spark SQL - Delta Lake Programming - Strong proficiency in: - Python - SQL Data Engineering - Experience with: - ETL / ELT Development - Data Modeling - Batch Processing - Streaming Data Processing Cloud Platform - Strong experience with GCP services such as: - BigQuery - Dataflow - Cloud Composer **Good-to-Have Skills** - Kafka - Event Hub - Airflow - Terraform - Docker - Kubernetes - GitHub Actions - Azure DevOps - Power BI - Tableau