Python-Pyspark Developer

Job description

Educational Requirements - Bachelor of Engineering - BTech - BCA - BSc - MTech - MSc - MCA Service Line Data Analytics Unit Responsibilities - We are looking for an experienced Python PySpark Developer to design, develop, and optimize large-scale data processing systems. - The ideal candidate will work on big data platforms, build scalable ETL pipelines, and process high-volume datasets using Spark and Python. - Develop and maintain data pipelines using Python and PySpark. - Process and transform large datasets in distributed environments. - Build scalable ETL/ELT workflows. - Work with Apache Spark (PySpark) for batch and real-time processing. - Optimize Spark jobs for performance and efficiency. - Handle structured and unstructured data. - Ingest data from multiple sources: Databases (SQL/NoSQL), APIs, Files (CSV, JSON, Parquet). - Integrate with data platforms like Hadoop (HDFS) and Cloud (AWS, Azure, GCP). - Tune Spark jobs (partitioning, caching, parallelism). - Optimize SQL queries and transformations. - Improve data processing efficiency and cost. - Collaborate with data engineers, data scientists, and analysts. - Translate business requirements into technical solutions. - Participate in code reviews and agile development practices. - Debug and resolve issues in data pipelines. - Monitor job execution and data quality. - Ensure reliability and availability of data workflows. Technical and Professional Requirements - Primary skills: Python, PySpark Preferred Skills - OpenSystem - Python - PySpark - Big Data - Data Processing