PySpark Developers / Leads

Job description

**Role & responsibilities** Experience 5 to 15 Years - Data Processing: Design, develop, and maintain data processing solutions using PySpark, Apache Spark, and Python. - Data Pipeline Development: Develop and optimize data pipelines using PySpark, Apache Spark, and cloud-based data platforms. - Data Integration: Integrate data from various sources, including relational databases, NoSQL databases, and cloud storage. - Data Transformation: Develop and implement data transformation logic using PySpark, Apache Spark, and Python. - Collaboration: Work with cross-functional teams to identify and prioritize project requirements, provide technical guidance, and ensure data quality. **Required Skills**: - PySpark: In-depth knowledge of PySpark, Apache Spark, and Python. - Data Processing: Strong understanding of data processing concepts, including data ingestion, data transformation, and data storage. - Cloud Experience: Experience with cloud-based data platforms, including AWS, Azure, or Google Cloud. - Expertise in DataFrames & Spark SQL, Spark Streaming with Apache Kafka real-time data ingestion pipelines - Very good conceptual understanding of Multithreading, distributed computing concepts of Pyspark - Programming: Proficiency in programming languages, including Python, Java, or Scala. - Communication: Excellent communication and collaboration skills.