Spark Scala Developer

Job description

Educational Requirements - Bachelor of Engineering - BTech - BCA - BSc - MTech - MCA - MSc Service Line Data Analytics Unit Responsibilities - Design and build scalable big data processing applications. - Strong experience in Apache Spark, Scala, and distributed data processing; work on building high-performance data pipelines for analytics and data engineering use cases. - Develop and maintain data pipelines using Apache Spark (Scala). - Process large-scale datasets in distributed environments. - Implement batch and real-time data processing solutions. - Write efficient and scalable code using Scala. - Work extensively with Spark Core, Spark SQL, and Spark Streaming. - Optimize Spark jobs for performance and resource utilization. - Ingest data from various sources: Databases (RDBMS, NoSQL), APIs, File systems (HDFS, S3). - Build ETL/ELT pipelines and data transformation workflows. - Tune Spark applications (partitioning, caching, memory tuning). - Optimize queries using Spark SQL. - Improve job execution time and reduce resource consumption. - Work with data engineers, data scientists, and business analysts. - Translate business requirements into scalable data solutions. - Participate in code reviews and architecture discussions. - Monitor data pipelines and Spark jobs. - Debug job failures and performance bottlenecks. - Ensure reliability and data quality in pipelines. Technical and Professional Requirements - Primary skills: Big Data - Data Processing - Spark; Functional Programming - Scala. Preferred Skills - Functional Programming - Scala - Big Data - Data Processing - Spark