AI Model Governance Specialist (Risk)

Job description

Job Summary The role will be responsible for independently evaluating AI/LLM systems, identifying potential risk vectors, designing robust evaluation frameworks, and establishing controls to support responsible and secure enterprise AI deployment. Roles & Responsibilities • Conduct RAG & GenAI risk assessments across end-to-end RAG pipelines, including knowledge-base ingestion, chunking, embeddings, vector retrieval, context grounding, and generation. • Identify AI/LLM risks such as data leakage, prompt injection, context contamination, retrieval failures, and hallucinations. • Design and execute LLM evaluation and benchmarking frameworks to assess hallucination rates, toxicity, robustness, model calibration, fairness, and bias. • Curate golden/reference datasets and design evaluation test cases with reference answers. • Implement and validate LLM-as-a-Judge evaluation pipelines. • Establish AI governance and risk mitigation controls covering data preprocessing, pre-deployment validation gates, and post-deployment observability. • Independently inspect, evaluate, and validate AI/LLM model outputs. • Conduct technical audits of AI/LLM systems and assess model performance against defined evaluation criteria. • Support the development of enterprise AI control frameworks covering data provenance, model sign-off, rate-limiting, and continuous drift monitoring. • Evaluate automated AI assessment and hallucination detection tooling, including Ragas, TruLens, and DeepEval. Qualifications • B.Tech / M.Tech in Computer Science, Quantitative disciplines, or MCA. • Strong quantitative and/or Computer Science foundation. • Thorough understanding of AI/ML taxonomy and modern Generative AI workflows. • Deep understanding of RAG architecture and associated risks, including vector search, embeddings, relevance scoring, and context grounding/grounding boundaries. • Hands-on familiarity with reference-based testing, LLM-as-a-Judge frameworks, red-teaming fundamentals, and semantic similarity evaluation. Experience & Skills 9 to 12 years of relevant experience Required Technical Skills: • Strong expertise in Generative AI / GenAI and Large Language Models (LLMs). • Strong understanding of Retrieval-Augmented Generation (RAG) pipelines. • Experience in LLM evaluation, model validation, AI risk assessment, and AI governance. • Understanding of AI risk vectors including prompt injection, hallucination, data leakage, and retrieval failures. • Familiarity with enterprise LLM orchestration and deployment platforms such as Amazon Bedrock, Azure OpenAI Service, and Vertex AI. Preferred / Good-to-Have Skills: • Working knowledge of Python for querying APIs, parsing JSON outputs, and independently auditing model evaluation scripts. • Practical experience with NLP evaluation metrics such as ROUGE, BLEU, BERTScore, and semantic distance metrics. • Ability to construct enterprise AI control frameworks covering data provenance, model sign-off, rate-limiting, and continuous drift monitoring. • Familiarity with automated hallucination detection and AI evaluation tools such as Ragas, TruLens, and DeepEval.