JobScans.in

Job description

Job Description - -------------- **Skill** : Python, Pyspark, AWS **Exp** : 6 to 12 years **Location** : Pune **Job Summary** We are seeking a skilled **Python PySpark AWS Developer** to design, develop, and maintain scalable data engineering solutions. The role involves building high-performance ETL pipelines, processing large datasets, and integrating with AWS cloud services to support enterprise analytics and reporting needs. **Key Responsibilities** * Design, develop, and optimize ETL pipelines using PySpark and Python. * Work with large-scale datasets to perform data cleansing, transformation, and aggregation. * Integrate with AWS services such as S3, Glue, EMR, Lambda, Redshift, Athena, Kinesis. * Implement data lake and data warehouse solutions on AWS. * Ensure data quality, consistency, and security across all processing stages. * Collaborate with data scientists, analysts, and business stakeholders to deliver data solutions. * Monitor, troubleshoot, and optimize data workflows for performance and cost efficiency. * Follow best practices for code versioning, CI/CD, and cloud resource management. * Write unit tests and maintain documentation for all developed solutions. **Required Skills \& Qualifications** * Strong programming skills in Python (data manipulation, APIs, automation). * Hands-on experience with PySpark for distributed data processing. * Proficiency in AWS cloud services (S3, Glue, EMR, Lambda, Redshift, Athena, IAM). * Solid understanding of SQL and relational database concepts. * Experience with data lake and data warehouse architectures. * Knowledge of ETL/ELT design patterns and best practices. * Familiarity with Git and CI/CD tools (e.g., Jenkins, GitHub Actions). * Strong problem-solving and debugging skills. * Bachelor's degree in Computer Science, Information Technology, or related field. **Preferred Skills** * Experience with streaming data (Kafka, Kinesis). * Knowledge of Docker and containerized deployments. * Familiarity with Terraform or CloudFormation for infrastructure as code. * Exposure to machine learning pipelines in a big data environment. * Understanding of data governance and compliance standards. **Work Environment** * Location: Onsite / Remote / Hybrid (as per company policy) * Team: Works closely with data engineers, analysts, and cloud architects. * Tools: AWS Cloud, Spark, Python, SQL, Git, CI/CD pipelines.