Data Migration + Databricks

Job description

**Job Description: Databricks DB Migration Engineer** Experience:6-8Years Locations: Pan India **Role Title** Databricks DB Migration Engineer **Experience** 68 years; 4+ years in data/database migration and 4+ years hands-on with Databricks preferred **Engagement Type** Full-time / Project-based **Primary Skills** Databricks, Apache Spark, PySpark, SQL, Delta Lake, ETL/ELT, Database Migration **Cloud Platforms** Azure preferred; AWS/GCP exposure added advantage **Reporting To** Data Platform Architect / Cloud Migration Lead / Delivery Manager **Role Overview** We are looking for a Databricks DB Migration Engineer responsible for planning, designing, executing, and validating database migration and modernization initiatives from legacy/on-premises or cloud data platforms to the Databricks Lakehouse platform. The role requires strong hands-on engineering capability in SQL, PySpark, Apache Spark, Delta Lake, data ingestion, performance tuning, reconciliation, and production cutover activities. **Key Responsibilities** - Assess existing source database environments, schemas, data volumes, dependencies, jobs, stored procedures, ETL pipelines, and reporting workloads for migration readiness. - Design migration approach for moving data and workloads from RDBMS, data warehouse, Hadoop, or cloud databases to Databricks Lakehouse using batch and incremental migration patterns. - Develop scalable ingestion and transformation pipelines using PySpark, Spark SQL, Databricks Workflows, Delta Live Tables, Auto Loader, and notebooks/jobs as applicable. - Convert and optimize SQL scripts, stored procedures, ETL logic, and business transformations into Spark SQL/PySpark-based implementations. - Implement Delta Lake best practices including partitioning, OPTIMIZE, Z-ORDER, schema evolution, time travel, ACID transactions, and data retention strategies. - Perform data profiling, cleansing, mapping, validation, reconciliation, and row/hash/count-level comparison between source and target systems. - Tune Spark jobs and Databricks clusters for performance, cost, concurrency, workload isolation, and operational efficiency. - Support migration dry runs, defect triage, production cutover, rollback planning, post-migration validation, hypercare, and operational handover. - Collaborate with architects, DBAs, data engineers, cloud teams, security teams, QA teams, and business stakeholders to ensure successful migration delivery. - Prepare technical documentation including migration design, mapping documents, runbooks, reconciliation reports, deployment plans, and operational support guides. **Mandatory Skills** - Strong hands-on experience with Databricks workspace, clusters, notebooks, jobs/workflows, Unity Catalog or workspace-level security controls. - Strong programming experience in PySpark and Spark SQL; Python scripting experience is required. - Good understanding of database migration lifecycle: assessment, planning, schema conversion, data migration, validation, cutover, and post-production support. - Experience working with source systems such as Oracle, SQL Server, Teradata, Netezza, PostgreSQL, MySQL, DB2, Hadoop/Hive, Snowflake, Redshift, Synapse, or BigQuery. - Good knowledge of relational database concepts, SQL tuning, query optimization, indexing concepts, schemas, constraints, and metadata analysis. - Hands-on exposure to ETL/ELT tools or frameworks such as Azure Data Factory, Informatica, Talend, SSIS, dbt, Airflow, or custom Spark-based pipelines. - Experience in data validation, reconciliation, data quality checks, and migration test automation. - Understanding of cloud storage and data formats such as ADLS Gen2/S3/GCS, Parquet, JSON, CSV, Avro, and Delta format. - Working knowledge of CI/CD, Git, release management, environment promotion, and deployment best practices. **Preferred Skills** - Databricks Certified Data Engineer Associate/Professional certification. - Experience with Azure Databricks, Azure Data Factory, ADLS Gen2, Azure SQL, Synapse, Event Hubs, Key Vault, and Azure DevOps. - Experience in migration accelerators, code conversion utilities, metadata-driven migration frameworks, or automated reconciliation frameworks. - Knowledge of Unity Catalog, data governance, lineage, access control, masking, and compliance requirements. - Exposure to medallion architecture, Lakehouse design patterns, CDC, streaming ingestion, and incremental processing. - Experience in Agile/Scrum delivery model and working with distributed teams. **Required Technical Competencies** - Databricks Lakehouse engineering and Delta Lake implementation. - Spark performance tuning including partition management, caching, shuffle optimization, file sizing, broadcast joins, and cluster sizing. - Schema migration, metadata analysis, data type mapping, and SQL-to-Spark conversion. - Batch migration, incremental migration, CDC-based migration, and large-volume data movement. - Data quality, validation, exception handling, audit logging, and operational monitoring. - Production support, job scheduling, failure recovery, alerting, SLA adherence, and incident resolution. Databricks certification is preferred. - Cloud certification in Azure/AWS/GCP is an added advantage. **Soft Skills** - Strong analytical and problem-solving skills with attention to data accuracy and migration quality. - Ability to communicate technical issues clearly to stakeholders, architects, and delivery leadership. - Strong ownership mindset with ability to work independently in high-pressure migration windows. - Good documentation discipline and ability to create reusable migration assets and runbooks. - Ability to collaborate with cross-functional teams across application, database, infrastructure, security, and business groups. **Deliverables / Success Metrics** - Successful migration of source data and workloads to Databricks within agreed timelines and quality thresholds. - Validated reconciliation reports wit