Data Engineer - Tester

Job description

Job Summary We are currently seeking a Data Engineer - Tester to join our team in Bengaluru, Karntaka (IN-KA), India (IN). Required Technical Skills • Data Testing Frameworks: Hands-on experience with DataTester to automate data quality checks. • Big Data Processing: Strong proficiency in PySpark and Apache Spark for manipulating and querying large-scale datasets. • Cloud Storage: Deep understanding of AWS S3 storage structures, partitioning strategies, and file formats (Parquet, Delta, Avro). • Testing Methodology: Solid background in functional and non-functional data testing, including data migration and regression testing. • Database & Querying: Advanced SQL skills for querying relational databases and verifying data warehousing solutions. • Version Control: Experience with Git and CI/CD deployment pipelines for automated test execution Key Responsibilities • Pipeline Validation: Build automated test suites to validate complex ETL/ELT pipelines using PySpark and DataTester. • Data Quality Assurance: Verify data accuracy, completeness, and consistency across source systems and AWS S3 data lakes. • Test Automation: Design and maintain scalable, reusable data testing frameworks to perform schema validation, null checks, and data reconciliation. • Component Testing: Conduct rigorous unit, integration, and regression testing on distributed data processing applications. • Defect Tracking: Identify, document, and track data anomalies, performance bottlenecks, and pipeline failures. • Collaboration: Partner closely with core Data Engineers and Platform teams to understand data models and implement test-driven development (TDD) practices.