Program Analyst Sr - Generative AI, Prompt Engineering & Validation

Job description

**Job Area:** Operations Group, Operations Group > Program Analyst General Summary: Assists with program development and implementation through managing processes, procedures, and tools that improve efficiencies. A Program Analyst coordinates across teams and monitors timelines, budgets, risks, and priorities to achieve program progress. Typically, a program needing a Program Analyst will be of significant size and will require expertise related to the development of project management mechanisms. Minimum Qualifications: Associate's degree in Business Administration, Management, Computer Science, Engineering, Computer Science, or related field. OR High School Diploma or equivalent and 2+ years of relevant work experience. **Job Title:** Program Analyst Sr Generative AI, Prompt Engineering Validation **Location:** Bangalore **Employment Type:** Full-time About the Role We are looking for a proactive and detail-oriented **Program Analyst** to support Generative AI initiatives focused on **prompt engineering, feature validation, test execution, output quality evaluation**, and program tracking. The role will support internal GenAI platforms and tools, including search, summarization, **RAG-based workflows**, analytical agents, and domain-specific AI assistants. The ideal candidate should combine strong program coordination skills with a practical understanding of GenAI systems, structured prompt-based testing, multi-source data validation, and clear reporting for engineering and leadership stakeholders. Key Responsibilities1. GenAI Program Tracking Coordination - Track GenAI feature validation plans, deliverables, owners, dependencies, and closure status across teams. - Maintain dashboards, trackers, action item logs, risk registers, and validation status reports. - Coordinate with engineering, product, platform, and domain expert teams to drive timely completion of testing and feedback cycles. - Consolidate weekly progress updates, blockers, risks, and recommendations for program reviews. - Ensure validation documentation, prompt libraries, and issue logs are accurate, current, and accessible. 2. Prompt Engineering Prompt Test Design - Create, refine, and maintain prompts for GenAI features, search workflows, summarization use cases, and analytical agents. - Develop zero-shot, few-shot, role-based, multi-turn, edge-case, and negative test prompts. - Build reusable prompt templates and feature-wise prompt libraries. - Compare prompt variants and document impact on accuracy, relevance, completeness, consistency, and latency. - Translate domain expert feedback into improved prompts and updated test scenarios. 3. Feature Validation Test Execution - Design and execute structured test cases for GenAI features, including functional, sanity, regression, negative, and exploratory testing. - Validate end-to-end behavior from user prompt to retrieved context, generated response, and final output format. - Map feature requirements to prompts, expected outputs, acceptance criteria, and validation evidence. - Identify defects, usability gaps, hallucinations, incomplete answers, inconsistent responses, and low-confidence outputs. - Support release readiness assessment using clear status indicators such as Go / Watch / Risk. 4. Data, Search RAG Validation - Validate AI outputs against structured, semi-structured, and unstructured data sources such as SharePoint, Confluence, Jira, documents, JSON/XML, and internal knowledge bases. - Check search relevance, context quality, source completeness, and data freshness before summarization. - Support validation of RAG-based workflows, semantic search, keyword search, global filters, and domain filters. - Identify data gaps, indexing issues, retrieval failures, and source-quality problems impacting AI response accuracy. - Maintain golden datasets, expected-answer sets, and benchmark queries for repeatable validation. 5. Output Quality Evaluation Defect Analysis - Evaluate GenAI responses for factual correctness, relevance, completeness, clarity, format quality, and consistency across repeated runs. - Log defects with clear reproduction steps, prompt used, expected output, actual output, severity, and owner. - Perform first-level root cause analysis to distinguish prompt issues, data issues, retrieval issues, model limitations, and product defects. - Work with SMEs to validate technical correctness and define acceptance thresholds for complex responses. - Track quality improvements over time and report trends in accuracy, hallucination reduction, and feature readiness. 6. Reporting, Documentation Stakeholder Communication - Prepare weekly validation summaries, risk updates, action item trackers, and executive-ready status reports. - Document prompt libraries, test suites, validation evidence, known issues, and improvement logs. - Facilitate review meetings, capture minutes of meeting, document decisions, and follow up on action items. - Communicate clearly with cross-functional stakeholders across engineering, platform, product, and program management teams. - Support leadership reviews with concise summaries, dashboards, and recommendations. 7. Automation Productivity Support - Use Excel, Power BI, Jira, Confluence, SharePoint, or similar tools to maintain trackers, dashboards, and reporting views. - Support basic scripting or automation for batch prompt execution, response capture, comparison, and reporting where applicable. - Help improve validation efficiency through reusable templates, structured checklists, and repeatable test processes. - Track productivity metrics such as test coverage, closure rate, defect aging, prompt reuse, and feature readiness. Required Skills Competencies - Hands-on exposure to Generative AI tools, LLM-based applications, prompt engineering, or AI-assisted workflows. - Strong ability to create structured test cases, validation reports, action trackers, and executive summaries. - Good understanding of prompt-