Senior AWS Engineer

Job description

**Job Description -** Senior AWS Engineer - Design and implement distributed data processing on AWS EMR using PySpark for batch/stream workloads. - Build robust, scalable ETL/ELT pipelines integrating diverse data sources (files, RDBMS, APIs, streaming). - Optimize Spark jobs (partitioning, caching, broadcast joins), and tune EMR clusters (instance types, autoscaling, bootstrap actions). - Implement data lake patterns on S3 with appropriate file formats (Parquet/ORC), schema evolution, and table management. - Orchestrate workflows using AWS Step Functions, Airflow, or EMR managed workflow, and automate CI/CD. - Ensure cost governance (spot instances, right-sizing, scaling policies) and observability (CloudWatch, Spark metrics). - Enforce security & compliance (IAM, KMS, encryption at rest/in transit, VPC, SGs, Glue Catalog access controls). - Collaborate with analytics teams to expose curated datasets to Redshift, Athena, Glue, or downstream BI tools. - Troubleshoot performance bottlenecks and production issues; implement resiliency patterns and SLAs. - Document architecture, standards, runbooks; mentor junior engineers.