AWS Data Engineer - G Noida

Job description

2. Key Responsibilities • Design, develop and maintain robust, scalable ETL/ELT pipelines on Databricks using Python, PySpark and Spark SQL, following medallion (Bronze/Silver/Gold) architecture principles. • Independently analyze business requirements, translate them into technical designs and solution options, and drive them to completion with minimal handholding. • Design and implement conceptual, logical and physical data models (dimensional and 3NF) for analytics, reporting and master data use cases. • Build and manage data ingestion from enterprise sources (SAP S/4HANA, Salesforce, flat files, APIs, streaming) into the Lakehouse. • Develop and operate workloads on AWS (S3, Lambda, IAM, networking fundamentals) integrated with the Databricks platform. • Implement data quality checks, validation rules, reconciliation and error-handling frameworks; prepare test scenarios and perform thorough unit and business-level validation of own deliverables before handover. • Apply Unity Catalog-based governance: access controls, lineage, cataloguing and aligned handling of personal data. • Optimize Spark jobs and Delta tables for performance and cost (partitioning, Z-ordering, cluster sizing, job orchestration). • Contribute to CI/CD practices for data pipelines (Git-based version control, code reviews, automated deployment) and produce clear technical documentation. • Support BI and MDM workstreams by delivering curated, well-modelled datasets for Power BI and Informatica IDMC consumption. • Provide L2/L3 support for deployed pipelines, troubleshoot production issues and drive root-cause resolution. 3. Primary (Must-Have) Skills • Python: Strong, hands-on programming for data engineering: clean, modular, well-tested code. • SQL: Advanced SQL for complex transformations, analysis and performance optimization. • Apache Spark / PySpark: Deep understanding of Spark architecture, Data frame API, Spark SQL, performance tuning and debugging of distributed jobs. • Data Modelling: Solid experience in dimensional modelling (star/snowflake), normalized modelling, slowly changing dimensions and canonical/master data models. • Databricks: Proven project experience with Databricks workspaces, Delta Lake, Delta Live Tables/Jobs, Workflows and Unity Catalog. • AWS: Working proficiency with core AWS data services (S3, Glue, Lambda, IAM, CloudWatch) and integration with Databricks. 4. Other Required Skills • Experience with orchestration tools (Databricks Workflows or equivalent). • Git-based development workflow, code review discipline and CI/CD for data pipelines (Azure DevOps or similar). • Exposure to integrating with enterprise systems such as SAP S/4HANA and Salesforce (APIs, CDC, extractors) is highly desirable. • Familiarity with Power BI datasets/semantic models and with MDM concepts (e.g., Informatica IDMC) is an advantage. • Exposure to streaming technologies (Structured Streaming, Kafka/Kinesis) is a plus. 5. Ways of Working & Behavioral Expectations • Independent delivery: Works independently end-to-end: clarifies requirements early, proposes designs, and delivers complete, tested solutions without repeated review cycles. • Ownership: Takes accountability for quality and timelines; proactively prepares test scenarios and validates business logic before submitting work. • Speed of comprehension: Grasps new requirements and domain logic quickly and converts them into working solutions at the pace the project demands. • Communication: Communicates progress, risks and blockers clearly and early; documents solutions to a standard others can maintain. • Teamwork: Collaborates effectively with BI developers, MDM consultants, business SMEs and vendor teams in a multi-vendor environment. 6. Qualifications • Bachelor's degree in Computer Science, Engineering or a related field (Master's preferred). • Relevant certifications are an advantage: Databricks Certified Data Engineer (Associate/Professional), AWS Certified Data Engineer / Solutions Architect.