SDE - II Data Platform

Job description

SDE II - Data Platform - Lucknow Mission Contribute to building a self-serve Data Platform-as-a-Product enabling scalable, reliable, and easy-to-use data infrastructure across batch and streaming workloads. Core Responsibilities • Build and maintain scalable data pipelines (batch & streaming) using Spark/Flink • Develop integrations across Airflow, Iceberg, and cloud platforms • Apply best practices in data modeling, partitioning, and query optimization • Support Kubernetes-based data platform workloads and troubleshoot issues • Optimize pipelines for performance, cost, and reliability • Implement monitoring, logging, and data quality checks • Participate in on-call and incident resolution Required Skills • 3–6 years in data/platform engineering • Strong fundamentals: SQL, data modeling, partitioning, optimization • Hands-on with Spark (PySpark/Scala); exposure to Flink is a plus • Experience with Airflow (or similar orchestration tools) • Familiarity with data lake formats (Iceberg/Delta/Hudi) • Working knowledge of Kubernetes & cloud (GCP/AWS/Azure) • Programming: Python (must-have), Scala (good-to-have) • Exposure to CI/CD, Git, and IaC (Terraform/Helm basics) Good to Have • Kafka / streaming systems experience • DBT, Great Expectations, or similar tools • Observability tools (Prometheus, Grafana) • Metadata/catalog tools (OpenMetadata/DataHub)