Walk-in || Aws Data Engineer - Walk-In

Job description

**Key Responsibilities** - Design and build **scalable data pipelines using Python and PySpark** - Develop and maintain **ETL/ELT workflows using AWS services (Redshift, Glue, EMR, S3, Lambda)** - Work with **structured and unstructured datasets at scale** - Ensure **data quality, reliability, and performance of pipelines** - Implement data processing for both **batch and near real-time pipelines** - Optimize SQL queries and data transformations - Collaborate with stakeholders (business, product, analytics teams) to understand data needs - Perform **data modelling, transformation, and validation** - Automate workflows using **Airflow or scheduling tools** - Improve data infrastructure for scalability and efficiency - Maintain clear documentation for all data components and pipelines **MustHave Skills** **Programming & Big Data** - Strong expertise in: - **Python** - **PySpark / Apache Spark** - Experience with: - Distributed data processing **AWS Data Platform** - Hands-on experience with: - **Redshift** - **Glue** - **EMR** - **S3** - **Lambda** - Strong experience working in **cloud-based data platforms** **Data Engineering** - Strong experience in: - **ETL/ELT pipelines** - **Data ingestion and transformation** - Experience handling **batch and near real-time pipelines** **Database & SQL** - Strong proficiency in: - **SQL (joins, aggregations, performance tuning)** - Understanding of: - OLTP and OLAP systems **Scheduling & Workflow** - Experience with: - **Airflow (preferred)** or other schedulers **Data Warehousing** - Strong experience with: - **AWS Redshift or Hive** - Understanding of: - Data warehousing and data lake concepts **GoodtoHave Skills** - Experience with: - **Hadoop ecosystem** - **NoSQL databases (DynamoDB / MongoDB)** - Exposure to: - **ETL tools (Informatica / Netezza)** - Knowledge of: - **Data modelling concepts** - Experience with: - AWS services like **Athena, Step Functions** - Domain knowledge: - **Investment Banking / Financial Services**