Director of Engineering

Job description

Experian is a global data and technology company, powering opportunities for people and businesses around the world. We operate across a range of markets, from financial services to healthcare, automotive, agribusiness, insurance, and many more. Experian invests in people and new advanced technologies to unlock the power of data. We have an amazing team of 25,200 people in 32 countries. Our uniqueness is that we celebrate yours. Experian's people first, inclusive and purpose driven culture is multi award-winning; World's Best Workplaces™ 2025 (Fortune Global Top 25), Great Place To Work™ in 26 countries to name a few. Check out Experian Life on social or explore our Careers Site (experian.com/careers) to understand why. We are seeking an Engineering Director with deep Data Engineering expertise to lead the design and delivery of a next-generation, AI-enabled data ingestion platform on AWS. You will be #LI-hybrid based in Hyderabad and reporting to CTO. This role owns the strategy, architecture, and execution for how data is ingested, transformed, modelled at scale — across both batch and real-time (streaming) pipelines. Data engineering is the core of this role. You will define the pipeline architecture, data models, and quality frameworks that everything else depends on, while modernising legacy and mainframe pipelines into cloud-native, agent-assisted systems. This is a hands-on leadership role that blends data engineering, platform engineering, and AI/DevOps excellence, responsible for building high-performing teams and delivering reliable, scalable, and intelligent data systems. Key Responsibilities Data Engineering & Pipeline Architecture (Core) • Own the end-to-end architecture for batch and streaming data ingestion — source onboarding, extraction, transformation, loading, and serving. • Design robust, scalable data models and schemas (dimensional, normalized, and data-vault where appropriate) for a modern lakehouse. • Build high-throughput, fault-tolerant pipelines using distributed processing (Apache Spark / EMR, Glue, Flink) with idempotency, replay, and backfill built in. • Architect real-time ingestion and change-data-capture (CDC) using Kafka / MSK / Kinesis and streaming frameworks. • Standardize open table and file formats (Apache Iceberg / Delta / Hudi, Parquet, Avro) and manage schema evolution and partitioning strategies. • Own pipeline orchestration and dependency management (Airflow / MWAA, Step Functions, dbt) with clear SLAs for freshness and latency. • Drive performance, scalability, and cost optimization across compute and storage (partition pruning, file compaction, resource tuning, FinOps). Data Quality, Governance & Reliability • Embed data quality, validation, lineage, and control frameworks directly into ingestion (Great Expectations / Deequ-style checks, data contracts). • Define and enforce data contracts, cataloging, and metadata management (AWS Glue Data Catalog, Lake Formation). • Ensure parity validation and controlled, phased migration from legacy and mainframe systems with zero data loss. • Implement observability for data pipelines — freshness, volume, schema-drift, and anomaly detection — with SLOs/SLIs and error budgets. • Ensure DevSecOps, encryption, access control, and compliance (PII handling, GDPR, data residency) across the platform. Strategy & Engineering Leadership • Define and execute the data-first, AI-enabled ingestion platform strategy aligned to business outcomes. • Build and lead teams spanning Data Engineering, Platform/DevOps, and AI/ML. • Establish engineering standards, reusable frameworks, operating models, and delivery roadmaps. • Mentor senior and staff engineers; grow technical depth and set the bar for engineering craftsmanshi. AI & Agentic Engineering • Introduce AI-assisted engineering (Claude Code, Copilot) across the data SDLC to accelerate pipeline development and testing. • Leverage AWS Bedrock / LLM platforms for ingestion intelligence — schema inference, mapping, validation, and anomaly detection. • Drive AIOps and agent-driven automation for auto-remediation, intelligent alerting, and self-healing pipelines. Platform, DevOps & Cloud Foundations • Establish best-in-class CI/CD, GitOps, and release engineering for data pipelines and platform components. • Standardize Infrastructure as Code (Terraform / CDK) and reusable, self-service platform building blocks. • Drive event-driven and microservices-based architectures on AWS-native services (S3, Glue, Lambda, EKS, MSK/Kinesis). Stakeholder & Delivery Management • Partner with Product, Architecture, Data Governance, and Business teams on roadmaps and prioritization. • Manage delivery, risk, and cross-program dependencies; communicate trade-offs clearly to senior stakeholders. &#xa0; • 12+ years in software/data engineering, with 5+ years in engineering leadership roles. • Proven experience leading data engineering, platform, or data-intensive organizations at scale. Data Engineering (Core) • Deep expertise designing and operating large-scale batch and streaming data pipelines in production. • Dstributed data processing: Apache Spark (EMR/Glue), and Flink or Beam. • Data modeling and warehousing/lakehouse design (Redshift, Snowflake, Databricks, Athena, or equivalent). • Lakehouse table formats and file formats: Iceberg / Delta / Hudi, Parquet, Avro; schema evolution and partitioning.<