Senior Spark Scala Developer

Job description

- Design and enhance Apache Iceberg table implementations and integrations. - Architect scalable lakehouse solutions on object stores (S3, ADLS, GCS, HDFS). - Implement and optimize metadata handling, schema/partition evolution, and time travel. - Improve query performance across engines (Spark, Flink, Trino, Presto). - Contribute to Apache Iceberg open-source development. - Analyze query execution plans and optimize read/write paths. - Collaborate on best practices for ingestion, governance, and lifecycle management. - Lead technical design discussions and resolve complex production issues. - Mentor engineers and conduct code reviews. Required education Bachelor's Degree Preferred education Master's Degree Required technical and professional expertise ; - Bachelors or Masters in Computer Science or related field. - 12+ years of software development experience. - 5+ years of hands-on experience with;Apache Spark and;Scala. - Strong knowledge of distributed computing and cluster frameworks. - Proficiency in;Scala and functional programming principles. - Expertise in Spark tuning, partitions, joins, and optimization techniques. - Experience with cloud platforms (AWS,;Azure,;GCP) and tools like;EMR,;Databricks,;HDInsight. - Familiarity with;Kafka,;Hive,;HBase, NoSQL databases, and data lake architectures. - Knowledge of;CI/CD,;Git,;Jenkins, and automated testing. - Strong problem-solving and collaboration skills. Preferred technical and professional experience - Experience with;Databricks,;Delta Lake, or;Apache Iceberg. - Experience in;Optimiser - Experience with;Gluten - velox - Exposure to machine learning pipelines using;Spark MLlib or integration with ML frameworks. - Open-source contributions in big data projects. - Excellent communication and leadership abilities. **Years of Experience:** 12;-;14