Senior Data Engineer

Job description

**. Functional Programming & Apache Spark** - **Scala Core Mastery** : You must understand functional programming paradigms, immutable data structures, pattern matching, and implicit parameters. - **Spark Core & Architecture** : Deep knowledge of the internal workings of Apache Spark, including the Catalyst Optimizer, Tungsten execution engine, lazy evaluation, Directed Acyclic Graphs (DAGs), and memory management (execution vs. storage memory). - **Performance Tuning** : Ability to identify and resolve performance bottlenecks like data skew, handling OOM (Out Of Memory) errors, optimizing joins (Broadcast vs. Sort-Merge), managing partition sizes, and avoiding expensive shuffle operations. - **Structured APIs & Streaming** : Proficiency in Spark DataFrames/Datasets APIs and Spark Structured Streaming for low-latency, real-time data processing. **2. Next-Generation Storage & Table Formats** - **Apache Iceberg** : Expertise in implementing Iceberg as your open table format over a data lake. You must master features like ACID transactions, time travel, schema evolution (hidden partitioning), and row-level updates/deletes. - **Apache Ozone** : Understanding Ozone as a scalable, redundant, and distributed object store designed specifically for Hadoop environments. You should know how it replaces or coexists with HDFS to handle billions of small and large files efficiently. - **Storage Optimization** : Skills in managing data compaction (merging small files), snapshot isolation, and choosing optimal file formats like Parquet, ORC, or Avro. **3. The Hadoop Ecosystem Foundation** - **HDFS & YARN** : While industry focus is shifting toward object storage, you still need a strong understanding of HDFS architecture (NameNode, DataNode) and YARN resource management (Resource Manager, Node Manager) to debug legacy systems or manage hybrid environments. - **Hive & Metastore Management** : Ability to manage catalog metadata and run distributed SQL queries over your distributed storage system. **4. Workflow Orchestration** - **Apache Airflow** : Mastery of building, scheduling, and monitoring complex data pipelines using Python-based DAGs. - **Advanced Airflow Concepts** : Utilizing TaskFlow API, custom XComs, dynamic task mapping, and setting up efficient Task Groups. - **Orchestration Integration** : Knowing how to safely trigger, monitor, and pass parameters to external Spark jobs or Cloud/Databricks operators) without overloading the Airflow worker nodes. **5. Architectural & Cross-Functional Skills** - **Data Lakehouse Architecture** : Designing unified platforms that combine the cost-effective storage of data lakes with the data management structures of data warehouses. - **CI/CD & DataOps** : Writing clean, testable Scala/Python code using unit-testing frameworks (like ScalaTest) and automating deployments using Git, Docker, and CI/CD pipelines. - **Advanced SQL** : Writing complex query logic, analytical window functions, and diagnosing execution plans—even when writing Spark code, SQL remains foundational.