Aws Data Engineer

Job description

Job Summary We are seeking an experienced Senior Data Engineer to design, build, and support scalable data integration and data processing solutions across cloud-based platforms. The ideal candidate will have strong expertise in ETL/ELT development, large-scale data processing, AWS services, workflow orchestration, and modern data architecture. This role requires close collaboration with cross-functional teams to deliver reliable, high-performance data pipelines that support business intelligence, analytics, and operational data needs. Key Responsibilities • Design, develop, and maintain scalable ETL/ELT pipelines for data ingestion, transformation, and integration. • Build and support data workflows across cloud-based environments and distributed data platforms. • Develop robust data solutions capable of processing large volumes of structured and unstructured data. • Create and optimize SQL queries, data models, and database solutions to ensure data quality and performance. • Implement orchestration and scheduling workflows using tools such as Apache Airflow, Azure Data Factory, or AWS Step Functions. • Design resilient and fault-tolerant data pipelines with logging, monitoring, retry mechanisms, idempotency, and recovery capabilities. • Work with data lakes, data warehouses, and relational/non-relational databases to support enterprise analytics. • Integrate APIs, streaming platforms, and event-driven architectures into data engineering solutions. • Collaborate with architects, software engineers, analysts, and business stakeholders to define and deliver data requirements. • Support DevOps practices including CI/CD pipelines, automated testing, containerization, and infrastructure-as-code. • Participate in code reviews, troubleshooting, performance tuning, and production support activities. • Ensure adherence to data governance, security, compliance, and quality standards. Required Qualifications • Bachelor's degree in Computer Science, Engineering, Information Systems, or a related technical discipline. Equivalent practical experience may be considered. • 5+ years of experience in Data Engineering, ETL/ELT development, or Data Integration. • Strong experience designing and supporting large-scale data processing solutions in cloud or distributed environments. • Advanced proficiency in SQL. • Hands-on experience in at least one programming language: • Python • Java • Scala • Spark SQL • Experience with workflow orchestration tools such as: • Apache Airflow • Azure Data Factory (ADF) • AWS Step Functions • Strong understanding of: • Relational databases • NoSQL databases • Data warehouses • Data lake architectures • Experience implementing monitoring, logging, error handling, retry logic, and recovery processes for production data pipelines. • Excellent analytical, problem-solving, communication, and collaboration skills. Preferred Qualifications • Experience working with AWS cloud data platforms. • Hands-on experience with: • AWS Lambda • Apache Kafka • Knowledge of: • Data Modeling • Metadata Management • Data Governance • Master Data Management (MDM) • Experience with: • Docker • Kubernetes • CI/CD pipelines • Infrastructure as Code (Terraform/CloudFormation) • Experience working in regulated industries such as: • Financial Services • Banking • Healthcare • Insurance Technical Environment The successful candidate will work within a modern data engineering ecosystem that may include: • AWS Cloud Services • Data Lakes and Data Warehouses • Distributed Compute Platforms • Apache Kafka and Streaming Systems • APIs and Data Integration Services • Apache Airflow / AWS Step Functions • Git and Source Control Management • Automated Testing Frameworks • Monitoring and Observability Platforms • Docker and Containerized Deployments • CI/CD and Infrastructure as Code Tools