Pyspark Developer

Job description

PySpark Developer **Experience:** 7+ Years **Location:** Pune & Chennai **Joining:** Immediate / 015 days preferred **Job Summary:** We are seeking a skilled PySpark Developer with 7+ years of experience in big data technologies. The ideal candidate should have strong expertise in PySpark, data processing, and building scalable data pipelines. You will be responsible for designing and optimizing large-scale data workflows in a distributed environment. **Key Responsibilities:** - Design, develop, and maintain scalable data pipelines using **PySpark** - Process and analyze large datasets in distributed environments - Work with **Apache Spark**, Hadoop ecosystem, and big data tools - Optimize data processing performance and ensure efficiency - Develop ETL/ELT workflows for data ingestion and transformation - Collaborate with data engineers, analysts, and stakeholders - Ensure data quality, consistency, and governance - Troubleshoot and resolve production issues in data pipelines - Document technical designs, processes, and workflows **Required Skills:** - Strong hands-on experience in **PySpark and Apache Spark** - Expertise in **Python programming** - Experience with **Hadoop ecosystem (HDFS, Hive, etc.)** - Strong SQL skills and data manipulation capabilities - Experience in building **ETL pipelines** - Knowledge of data warehousing concepts - Familiarity with distributed computing and optimization techniques **Preferred Skills:** - Experience with **Cloud platforms (AWS / Azure / GCP)** - Knowledge of **Databricks** or similar platforms - Experience with **Airflow, Kafka, or streaming technologies** - Exposure to **Delta Lake or Lakehouse architecture** - Understanding of **CI/CD pipelines** and DevOps practices **Qualifications:** - Bachelor’s or Master’s degree in Computer Science, IT, or related field - Relevant certifications in Big Data or Cloud technologies are a plus