Python, Spark Scala Developer

Job description

Educational Requirements Bachelor of Engineering Service Line Data Analytics Unit Responsibilities - Data Engineering DevelopmentDevelop and maintain batch/stream data processing jobs using Spark with Scala and/or Python.Build reusable components and utilities to standardize ingestion, transformation, and validation patterns. - Implement efficient transformations, joins, aggregations, and partitioning strategies for large-scale datasets. - Performance, Quality ReliabilityTune Spark jobs for performance (e.g., caching, shuffles, partition sizing) and troubleshoot failures or bottlenecks. - Write unit/integration tests and ensure code quality through reviews, refactoring, and documentation. - Monitor job runs, analyze logs/metrics, and drive continuous improvements in stability and runtime. - Data SQLUse SQL to validate data, reconcile outputs, and support downstream reporting/analytics needs. - Collaborate with stakeholders to clarify requirements and translate them into robust data processing logic. Additional Responsibilities: - Bachelors degree in Computer Science, Engineering, or a related field. - 25 years of hands-on experience building solutions with Python. - 25 years of experience working with Apache Spark for large-scale data processing. - 25 years of experience developing with Scala (or strong ability to work across Scala and Python codebases). - Working knowledge of SQL for querying, validation, and troubleshooting datasets. - Strong problem-solving skills and ability to collaborate effectively in a team environment. Preferred Skills: - Technology->Analytics - Packages->Python - Big Data - Technology->Functional Programming->Scala - Technology->Big Data - Data Processing->Spark