Databricks ETL Testing - 10th July (Friday) - Virtual Drive

Job description

ETL Databricks Testing Required Technical Skill Set : Strong experience in ETL testing and data warehouse concepts. • Hands-on expertise in Databricks (including notebooks, clusters, and jobs). • Proficiency in SQL for data validation and analysis. • Experience with big data technologies (Spark, PySpark, Delta Lake). • Familiarity with cloud platforms (Azure, AWS, or GCP) and their data services. • Knowledge of data modeling, data quality frameworks, and BI tools. • Strong analytical and problem-solving skills. • Excellent communication and documentation abilities. Must Have: - ETL / Data Warehouse Testing: o Strong experience in DWH concepts (Facts, Dimensions, Star/Snowflake schema) o Hands-on data validation across large datasets - Databricks: o Experience validating data pipelines built on Databricks o Working knowledge of notebooks and Delta Lake concepts - SQL: o Strong proficiency in writing complex SQL queries (joins, subqueries, aggregations, window functions) - Ability to perform large-volume data validation and reconciliation - Python (Basic to Intermediate): o Ability to read/write simple Python scripts for data validation, automation support, or log analysis Responsibility : 1 Validate end to end ETL/data pipeline workflows from source systems to target data warehouse layers 2 Perform data reconciliation, data quality checks, and transformation validation across Bronze / Silver / Gold layers (or equivalent) 3 Execute Databricks-based data validations using SQL, notebooks, and result comparisons 4 Design and execute complex SQL queries for data profiling, completeness, accuracy, and referential integrity checks 5 Validate Delta tables, fact/dimension tables, aggregates, and historical load logic 6 Perform job failure analysis, data discrepancy RCA, and support defect triage 7 Collaborate closely with Data Engineering, Analytics, and QA teams during releases 8 Ensure test coverage for incremental loads, CDC, reprocessing, and batch schedules