AWS Data Engineer- Airflow

Job description

Senior Data Engineer JD Design develop and maintain scalable ETL pipelines using AWS Glue PySpark and AWS Lambda Migrate structured and semistructured data from existing data lakes eg S3 Hadoop to Amazon Redshift Optimize data ingestion and transformation processes for performance and costefficiency Collaborate with data architects analysts and business stakeholders to understand data requirements and ensure data quality Implement data validation cleansing and transformation logic Monitor and troubleshoot data pipelines and workflows Ensure data security and compliance with organizational and regulatory standards Document data flows architecture and processes Responsibility Design develop and maintain scalable ETL pipelines using AWS Glue PySpark and AWS Lambda Migrate structured and semistructured data from existing data lakes eg S3 Hadoop to Amazon Redshift Optimize data ingestion and transformation processes for performance and costefficiency **Primary Skills** AWS Glue jobs AWS services Lambda S3 AWS database Hadoop and Redshift Communication Usage of AI code assist tools Working knowledge of implementing data migration