Python+Spark Scala Professional

Job description

Educational Requirements - Bachelor of Engineering, BSc, BTech, BCA, MTech, MCA, MSc Service Line Data Analytics Unit Responsibilities - Big Data Spark Development - Develop and maintain data processing pipelines using Apache Spark (PySpark Scala) - Work with Spark DataFrames, RDDs, and Spark SQL - Implement transformations, joins, aggregations, and optimizations - Tune Spark jobs for performance, scalability, and reliability - Python Scala Programming - Write clean, efficient, and scalable code in Python and Scala - Develop modular and reusable components - Integrate data pipelines with various applications and APIs - ETL Data Engineering - Design and build ETL workflows for structured and unstructured data - Extract data from multiple sources (databases, APIs, flat - Perform data cleansing, transformation, and validation - Ensure data accuracy, consistency, and completeness - Data Platforms Integration - Work with Hadoop ecosystem (HDFS, Hive, Spark) - Handle large datasets in data lakes and warehouses - Process data in formats like Parquet, ORC, JSON, CSVC - Collaboration Support - Work with data engineers, analysts, and business stakeholders - Troubleshoot pipeline issues and provide production support - Participate in Agile/Scrum processes - Maintain technical documentation Additional Responsibilities - Core Skills - 25 years of experience in Python development - Hands-on experience with Apache Spark (PySpark and/or Scala) - Strong understanding of data processing and ETL concepts - Good knowledge of SQL and relational databases Technical and Professional Requirements - Primary skills:Technology- >Big Data - Data Processing- >Spark,Technology- >Java- >Apache,Technology- >Machine Learning- >Python Preferred Skills - Technology- >Java- >Apache- >Scala - Technology- >Big Data - Data Processing- >Spark- >SparkSQL - Technology- >AI-Data science- >PYTHON