Distinguished Data Engineer

Job description

About the Opportunity Job Type: PermanentApplication Deadline: 24 September 2026 Job Description Title Distinguished Data Engineer Department AMP Location Bengaluru Reports To Research & Sustainable Investing Data Engineering Lead Level 5 We share a commitment to making things better for clients and each other. We continually explore new technology and different ways of working to put our clients first. Bring your boldest ideas to our Research & Sustainable Investing Technology team and feel like you are making progress. About your team AMP Delivery is responsible for the design and delivery of all changes in business process and/or technology solutions that support the growth for Fidelity’s Global Investment Solutions & Services business. We partner with Investment Management, Asset Management Operations and Distribution teams across London, Hong Kong, Tokyo, Toronto, Australia, Singapore, and China. The Research & Sustainability team delivers strategic initiatives that enhance investment decision-making through modern research workflows, investment data platforms, sustainability capabilities, advanced analytics, and AI-powered solutions. We are increasingly leveraging Large Language Models (LLMs), frontier AI models, and agentic workflows to transform how investment professionals discover insights, conduct research, and make investment decisions across Equities, Fixed Income, and Multi-Asset. About your role This is a specialist, hands-on AI data engineering role within the AMP - Research & Sustainable Investing team. Reporting to the Data Engineering Lead, you will build the trusted data foundations that power machine learning, Large Language Model (LLM), retrieval-augmented generation (RAG), search and agentic AI workflows for investment research and sustainability use cases. You will engineer structured, semi-structured and unstructured data through the full AI data lifecycle - ingestion, transformation, enrichment, feature and evaluation dataset creation, embedding generation, vector indexing, retrieval and governed delivery. You will work primarily across Snowflake, AWS and Kafka while integrating with enterprise sources including Oracle and Microsoft SQL Server. You will work closely with AI/ML engineers, data engineers, research analysts, sustainability specialists and investment teams to make AI data accurate, discoverable, traceable, secure and production-ready. The role directly contributes to Accelerated Innovation, Cost Optimisation, Risk Mitigation and Business Enablement. Key Responsibilities • Design, build, test, deploy and operate production-grade data pipelines for structured, semi-structured and unstructured research, sustainability and investment data. • Build AI-ready datasets for machine learning and generative AI, including feature, training and evaluation datasets, embeddings, vector indexes and retrieval-augmented generation workflows. • Engineer retrieval data flows that curate, enrich, index and serve trusted content for semantic search, vector search and LLM-based applications, with appropriate metadata and provenance. • Help to develop metadata and knowledge structures, including taxonomies, ontologies, entity resolution and knowledge graphs where appropriate, to improve data discovery, retrieval and contextual understanding. • Build reusable data services and APIs that expose governed investment data to AI/ML models, applications, analytics and agentic workflows. • Apply data quality, lineage, data contracts, security, privacy, entitlements, auditability and provenance controls across data pipelines and AI-ready data products. • Use Snowflake, AWS, Kafka and enterprise data platforms to ingest and prepare data, integrating safely with Oracle and Microsoft SQL Server where source or legacy data is required. • Apply strong software and data engineering practices to data pipelines, including Python/SQL development, Git, CI/CD, automated testing, observability, monitoring, performance tuning and secure production support. • Partner with AI/ML engineers and investment stakeholders to translate AI use cases into reusable data products, trusted retrieval layers and measurable production outcomes. • Investigate complex data, embedding, indexing, retrieval and pipeline issues, perform root-cause analysis and implement sustainable fixes with the Data Engineering Lead and wider team. About You Core Technical Skills • Data Engineering: Strong hands-on experience preparing structured, semi-structured and unstructured data for machine learning and generative AI, including feature, training and evaluation datasets. • Embeddings, Vector Search and RAG: Practical experience building embedding pipelines, vector indexes, retrieval workflows and RAG data foundations for applications. • Evaluation Data: Experience creating and curating evaluation datasets and applying data-quality, relevance, traceability and provenance checks to support reliable retrieval and model evaluation. • Data Pipelines: Strong experience designing production-grade ingestion, transformation, enrichment and delivery pipelines using batch, near-real-time, streaming, API and event-based patterns. • Programming: Strong proficiency in Python, Spark and SQL, with the ability to build maintainable, tested production code for data preparation, embeddings, indexing, retrieval and data services. • Data Platforms and Integration: Experience with Snowflake, AWS and Kafka, plus enterprise relational sources such as Oracle or Microsoft SQL Server; able to integrate modern data and retrieval workloads with existing platforms. • Data Services and Orchestration: Experience building reusable APIs and data services and using orchestration tools such as Airflow or Control-M to operate dependable production data workflows. • Engineering Practices: Experience with Git, CI/CD, automated testing, infrastructure as code, data