QA Engineer (C1) - Agentic AI

Job description

**Job Description: Key Responsibilities** - **1. Agentic AI Testing, Evaluation & Automation** - Define and execute testing strategies for LLM-based, multi-agent, RAG, and Agentic AI systems. - Validate autonomous agent behavior, reasoning, memory, tool usage, API/database integrations, and end-to-end workflows. - Evaluate AI outputs for accuracy, relevance, groundedness, consistency, completeness, toxicity, bias, hallucination risk, and guardrail compliance. - Define AI quality KPIs such as hallucination rate, groundedness score, agent success rate, task completion rate, response relevancy, latency, cost efficiency, and user satisfaction. - Build automated evaluation pipelines, quality scoring mechanisms, dashboards, and CI/CD-integrated quality gates. - Develop reusable test harnesses, simulators, and benchmarking frameworks to compare models, prompts, and agent configurations. - **2. Team Leadership & Capability Building** - Build and lead a team of **Agentic AI Quality Engineers** . - Define team structure, testing standards, best practices, and governance models. - Mentor QA engineers in AI testing methodologies, evaluation techniques, and automation frameworks. - Drive innovation and adoption of emerging AI testing tools and technologies. - Collaborate with Product, Engineering, Data Science, and AI Research teams to improve overall AI quality. - **3. Reporting & Stakeholder Management** - Provide quality assessments and recommendations to leadership and stakeholders. - Present testing outcomes, risk assessments, KPI trends, and model evaluation reports. - Drive quality governance for Agentic AI initiatives across the organization. - Ensure traceability of testing activities, evaluation criteria, and quality benchmarks. **Responsibilities: Required Skills & Experience** **Technical Skills** - 7–12 years of experience in Software Testing, Quality Engineering, or Test Automation. - Minimum 3+ years of hands-on experience in GenAI, LLM Testing, Agentic AI Testing, or AI Quality Engineering. - Strong understanding of LLMs, AI agents, RAG, prompt validation, tool calling, agent memory, MCP, and multi-agent orchestration. - Experience defining AI quality metrics, evaluation methodologies, benchmarking frameworks, and model comparison approaches. - Hands-on automation experience with Python, Playwright, Pytest, API automation, test framework development, and CI/CD quality gates. - Experience with AI evaluation frameworks such as DeepEval, Ragas, LangSmith, OpenAI Evals, or equivalent tools. - Exposure to cloud platforms such as Azure, AWS, or GCP. **Soft Skills** - Strong communication and stakeholder management skills. - Analytical mindset with strong problem-solving ability. - Self-driven, outcome-oriented, and capable of leading multiple initiatives in a fast-evolving AI ecosystem. - Qualifications: Experience testing enterprise Agentic AI platforms and autonomous AI systems. - Hands-on exposure to frameworks or tools such as LangGraph, CrewAI, AutoGen, Semantic Kernel, Microsoft Copilot Studio, TruLens, or Promptfoo. - Exposure to AI observability, monitoring, model governance, responsible AI, and AI safety practices. - Experience building AI quality dashboards and KPI reporting systems. - ISTQB, AI Testing, GenAI, Azure AI, AWS AI, or equivalent certifications