Lead AI Data Engineer

Job description

**Job Description: Key Responsibilities** **1. Solution Architecture & Technical Leadership** - Architect **enterprise-grade agentic and LLM solutions** (single-agent, multi-agent, tool-driven workflows) - Define **scalable GenAI system design patterns** (RAG, orchestration layers, evaluation frameworks) - Act as the **technical anchor** for GenAI initiatives across projects - Drive **design reviews, architecture governance, and best practices** **2. Agentic AI & LLM Engineering** - Design and build **agentic systems using LLMs** for use cases such as: - Knowledge assistants - Document automation & intelligence - Workflow orchestration - Implement **advanced prompt engineering strategies** , prompt orchestration, and reasoning chains - Build **tool-calling / function-calling frameworks** for agent workflows **3. RAG & Retrieval Systems** - Lead end-to-end implementation of **RAG pipelines** : Data ingestion chunking embeddings vector indexing retrieval - response generation - Optimise retrieval quality (recall, relevance, grounding) - Evaluate and benchmark different architectures **4. Productisation & Engineering Excellence** - Develop **production-grade APIs/services** (FastAPI, Flask, etc.) - Drive **code quality, testing standards, and reusable architecture components** - Ensure solutions are **performance optimised (latency, cost, reliability)** **5. Governance, Safety & Evaluation** - Implement **LLM guardrails** : - Hallucination control - Safety filters - Policy enforcement - Define **evaluation frameworks** : - Response quality metrics - RAG benchmarking - Human-in-the-loop validation **6. Collaboration & Delivery Leadership** Partner with: Data Engineering - - pipelines, data quality, governance MLOps- deployment, CI/CD, monitoring Business/Product- use-case alignment - Drive **end-to-end delivery ownership** across multiple projects **7. Technical Leadership Responsibilities (Critical Addition)** - Mentor and guide **junior engineers and project teams** - Conduct **technical reviews, solution walkthroughs, and code reviews** - Support **pre-sales / RFPs / solution proposals** with architecture inputs - Drive **reusable accelerators, frameworks, and COE assets** - Stay ahead of industry evolution and help **shape EXL’s GenAI strategy** - Influence **technology choice, design decisions, and roadmap planning** **Must-Have Skills** **Experience** - **9–12 years total experience** - **2–4+ years hands-on in LLM / GenAI delivery (production use cases)** **LLM / GenAI & Agentic Engineering** - Strong hands-on experience with: - LLMs (Claude, OpenAI, etc.) - RAG pipelines and retrieval optimisation - GPT + Agentic AI implementation experience - Experience with: - LangChain, LangGraph, or similar frameworks - Agent orchestration and tool-calling architectures Deep understanding of: LLM limitations, evaluation, and optimisation strategies **Core Engineering** - Strong Python/Pyspark engineering expertise (production-grade development) with proven API integration experience - Deep data analysis experience and handling large volume of data - Fabric/Azure Databricks/Snowflake data engineering integration skills - Good exposure to: - Cloud platforms (Azure/AWS/GCP) - SQL Containers, CI/CD, monitoring **Data / AI Foundations (Mandatory)** Prior experience in one or more: - Data Engineering (ETL/ELT, pipelines, orchestration) - Data Science / ML lifecycle (especially NLP) Analytics engineering / data products **Leadership Capabilities** - Experience **leading solution design or small teams** - Ability to **translate business problems into AI solutions** Strong stakeholder communication and influencing skills **Good-to-Have / Preferred** - Fine-tuning approaches: **LoRA / PEFT / prompt tuning** - Experience with **Azure AI stack (Azure OpenAI, AI Search)** - Exposure to: - **Enterprise security & data privacy in GenAI** - **Coding agents / autonomous agent frameworks** - Experience in **insurance / BFSI domains** (valuable for EXL use cases) **Responsibilities: Key Responsibilities** **1. Solution Architecture & Technical Leadership** - Architect **enterprise-grade agentic and LLM solutions** (single-agent, multi-agent, tool-driven workflows) - Define **scalable GenAI system design patterns** (RAG, orchestration layers, evaluation frameworks) - Act as the **technical anchor** for GenAI initiatives across projects - Drive **design reviews, architecture governance, and best practices** **2. Agentic AI & LLM Engineering** - Design and build **agentic systems using LLMs** for use cases such as: - Knowledge assistants - Document automation & intelligence - Workflow orchestration - Implement **advanced prompt engineering strategies** , prompt orchestration, and reasoning chains - Build **tool-calling / function-calling frameworks** for agent workflows **3. RAG & Retrieval Systems** - Lead end-to-end implementation of **RAG pipelines** : Data ingestion chunking embeddings vector indexing retrieval - response generation - Optimise retrieval quality (recall, relevance, grounding) - Evaluate and benchmark different architectures **4. Productisation & Engineering Excellence** - Develop **production-grade APIs/services** (FastAPI, Flask, etc.) - Drive **code quality, testing standards, and reusable architecture components** - Ensure solutions are **performance optimised (latency, cost, reliability)** **5. Governance, Safety & Evaluation** - Implement **LLM guardrails** : - Hallucination control - Safety filters - Policy enforcement - Define **evaluation frameworks** : - Response quality metrics - RAG benchmarking - Human-in-the-loop validation **6. Collaboration & Delivery Leadership** Partner with: Data Engineering - - pipelines, data quality, governance MLOps- deployment, CI/CD, monitoring Business/Product- use-case alignment - Drive **end-to-end delivery ownership** across multip