Lead Data Scientist

Job description

About the Role: Grade Level (for internal use): 11  Team: As a lead data scientist in the EDO, Collection Platforms & AI – Cognitive Engineering team, you will own the technical direction of a large-scale entity resolution and linking  platform used across S&P Global. This platform combines classical ML and GenAI techniques to power intelligent matching and linking services across multiple data domains, exposed as  production APIs and orchestrated through downstream workflow systems. You will set technical strategy, champion the adoption of LLMs and agentic AI across the platform, architect its  evolution, mentor senior and junior data scientists, and be the final technical authority on design decisions across the codebase. You will work in a truly global team and be expected to  drive thoughtful risk-taking, cross-team alignment, and technical/AI innovation. What's in it for you:  - Own the technical vision for an enterprise-scale AI/ML platform used across S&P Global  - Be the org's go-to authority on LLMs and agentic AI, with room to prototype and productionize novel approaches  - Lead, grow, and set technical direction for a highly skilled team of data scientists and engineers  - Architect solutions to the highest-complexity, highest-impact problems in the org, end-to-end  - Have final say on architecture, tooling, and technical trade-offs for a multi-domain production system Responsibilities:   • Own the end-to-end technical architecture of the platform, including its API service layer, asynchronous distributed task orchestration, and multi-domain model registry  - Set the AI/ML roadmap across both classical ML (gradient boosting, embedding-based similarity, fuzzy/probabilistic matching) and GenAI approaches (LLM-based extraction, prompt engineering,  fine-tuning/customization, agentic and multi-agent workflows)  - Champion innovation with agentic workflows: design, prototype, and productionize multi-step, tool-using LLM agents that can plan, reason, call tools/APIs, and collaborate with other agents  to solve complex linking problems  - Evaluate and drive adoption of state-of-the-art agentic AI infrastructure, including MCP (Model Context Protocol) servers for tool/context integration, agent orchestration frameworks, agent  memory/state management, and retrieval-augmented generation (RAG) with vector stores  - Architect and oversee productionization of models and pipelines — packaging, versioning, deployment to containerized/orchestrated environments , and integration with  cloud infrastructure (object storage, secrets/parameter management)  - Define and enforce MLOps and LLMOps standards: CI/CD for models and prompts, experiment tracking, model/prompt/agent versioning, automated evaluation, and monitoring for both synchronous  services and background task workers  - Lead technical design reviews and code reviews across the codebase; set and enforce coding standards, testing discipline, and architectural consistency across all modules  - Drive build-vs-buy and fine-tune-vs-prompt-vs-agent decisions across LLM providers, embedding models, and agentic frameworks used across the platform  - Continuously scout the LLM/agentic AI landscape (new models, protocols, and tooling) and run rapid proof-of-concepts to assess production fit  - Own production reliability of deployed services — diagnosing and resolving issues across API, async tasks, databases, caching, and LLM/agent layers  - Mentor senior and junior data scientists on both classical ML and modern LLM/agentic AI techniques; run technical onboarding for new team members joining the platform  - Manage stakeholders across engineering, workflow orchestration teams, and business domain owners to align on roadmap and delivery timelines  - Represent the team's technical decisions and AI innovation initiatives to senior leadership and influence org-level AI tooling and platform standards   Technical Requirements:   • Deep, hands-on experience architecting production systems combining classical ML (gradient boosting: LightGBM/XGBoost, scikit-learn) with GenAI (LLMs from providers such as Gemini, OpenAI,  Anthropic; prompt engineering; fine-tuning/customization; embedding-based retrieval via sentence-transformers/Hugging Face)  - Proven experience designing and shipping LLM-powered agents and agentic workflows — planning/reasoning loops, tool use, task decomposition, and multi-agent collaboration — beyond  single-turn prompting  - Working knowledge of MCP (Model Context Protocol) and other emerging standards for connecting LLMs/agents to tools, data sources, and context; experience integrating or building MCP servers  - Understanding of RAG architectures and vector retrieval systems, and when to apply them versus fine-tuning, agentic tool-use, or classical search  - Familiarity with agent orchestration frameworks (e.g., Google ADK, or similar)  - Expert proficiency in Python and its data/ML ecosystem (Pandas, NumPy, PyTorch/TF, Transformers, scikit-learn) at a level sufficient to review and set standards for a large, multi-module  production codebase  - Strong understanding of asynchronous, distributed task architectures (task queues, message brokers) and how they interact with ML/LLM inference at scale  - Experience architecting and operating web API services (e.g., FastAPI or comparable) in production, including containerization and orchestration  - Working knowledge of cloud infrastructure (object storage, secrets/parameter management) and relational databases as used in production ML services  - Deep understanding of entity r