DE&A -Data Engineer ETL Developer (Snowflake)

Job description

We are seeking a **hands on** **QA Lead** to drive quality assurance for a **scalable, enterprise-wide data platform** for an insurance client. The role involves validating **batch and real-time data pipelines** , ensuring **data accuracy across Raw, Silver, and Gold layers** , and supporting **report rationalization and self-service analytics (Power BI)** . The QA Lead will define and implement **end-to-end data testing strategies** , covering ingestion (AWS Glue, Kinesis), transformation (DBT), and consumption layers, while ensuring **data quality, integrity, and performance optimization** . **Key Responsibilities** - QA Strategy & Leadership - Define and implement end-to-end QA strategy for the enterprise data platform - Establish test frameworks, standards, and governance for data validation - Lead QA planning, estimation, and execution across multiple data streams - Data Validation & Testing - Validate data across Raw, Silver, and Gold layers ensuring accuracy, completeness, and consistency - Perform source-to-target reconciliation for batch and real-time pipelines - Design and execute: - Data quality checks - Transformation validation (DBT models) - Aggregation and KPI validation - Batch & Real-Time Pipeline Testing - Test batch ingestion pipelines using AWS Glue - Validate real-time streaming data pipelines using Amazon Kinesis - Ensure data latency, sequencing, and event consistency in streaming pipelines - Reporting & Rationalization QA - Validate datasets powering Power BI self-service reports - Support report rationalization initiatives by ensuring consistency of KPIs and eliminating redundant data sources - Perform report/data reconciliation testing across legacy vs new platform - Automation & Tools - Develop and implement automated data testing frameworks - Leverage SQL, Python, and testing tools (e.g., Great Expectations, DBT tests, custom frameworks) - Enable continuous testing integration within CI/CD pipelines - Performance & Optimization Testing - Validate performance of: - Data pipelines - Queries in Snowflake - Identify bottlenecks and work with engineering teams to optimize pipelines and queries - Ensure scalability for large data volumes and concurrent workloads - Data Quality & Governance - Define and enforce data quality rules, thresholds, and monitoring - Implement data anomaly detection and alerting mechanisms - Ensure compliance with audit, reconciliation, and governance standards **Required Skills & Experience** **Core Technical Skills** - Strong experience in data testing / ETL testing / data QA - Hands-on expertise with: - Snowflake (data validation, SQL testing) - DBT (testing, model validation) - AWS Glue (batch pipeline validation) - Amazon Kinesis (real-time pipeline testing) - Advanced proficiency in SQL for data validation and reconciliation - Programming skills in Python (preferred) **Testing Expertise** - Experience in: - Data reconciliation (source vs target) - Data quality frameworks and validation techniques - Automated data testing tools - Understanding of medallion architecture (Raw, Silver, Gold layers) **Analytics & Reporting** - Experience validating Power BI reports and datasets - Strong understanding of business KPIs and reporting consistency **Domain Expertise (Preferred)** - Experience in Insurance domain (Policy, Claims, Billing data) - Familiarity with regulatory reporting, audit, and reconciliation requirements **Experience** - 8–12 years in QA / Data Testing / ETL Testing - 3+ years in QA leadership or lead role - Experience working on enterprise-scale data platforms