Software Quality Engineer  Conversational AI / LLM Testing

Job description

Software Quality Engineer – Conversational AI / LLM Testing Experience: 4-9 Years Location : Noida / Kolkata Domain : Airlines / Travel Technology Job Summary We are seeking a highly skilled Software Quality Engineer with hands-on experience in testing Conversational AI, LLM-based applications, chatbots, virtual assistants, and Agentic AI systems. The ideal candidate will have a strong automation and quality engineering background, deep understanding of how modern conversational systems operate, and solid experience within the Airline or Travel domain, particularly around Booking, Post-Booking, and Check-In workflows. This role involves validating AI-driven customer conversation journeys, ensuring response quality, evaluating non-deterministic outputs, and establishing robust testing strategies for intelligent systems. Key Responsibilities • Own and drive the quality strategy for conversational AI products and customer-facing virtual assistants. • Design, execute, and automate test scenarios for LLM-powered applications, chatbots, voice assistants, RAG-based systems, and agentic workflows. • Validate conversational flows across booking, post-booking servicing, and check-in experiences within airline and travel ecosystems. • Develop and maintain automated test frameworks using Python. • Create test datasets, golden datasets, and evaluation benchmarks to measure AI system performance. • Design evaluation mechanisms for non-deterministic AI outputs using: • Rubric-based scoring • LLM-as-a-Judge approaches • Human review workflows • Statistical quality assessment methods • Validate prompt behavior, context retention, retrieval accuracy, tool/function calling, and response consistency. • Identify conversational AI failure modes such as hallucinations, context loss, prompt injections, retrieval issues, and reasoning errors. • Collaborate with Product Managers, Data Scientists, AI Engineers, and Developers to improve system quality and reliability. • Participate in release validation, regression testing, performance testing, and production quality monitoring.  Mandatory Skills & Qualifications Quality Engineering • 4+ years of experience in Software Quality Engineering, QA Automation, or SDET roles. • Proven experience owning the quality of a product or platform from testing strategy to production validation. • Strong understanding of software testing methodologies, automation frameworks, and quality metrics.  Python Automation • Strong scripting and automation expertise in Python. • Experience developing automated test suites, validation scripts, and data-driven testing frameworks.  Conversational AI / LLM Testing Hands-on experience testing one or more of the following: • Large Language Model (LLM) applications • Conversational AI platforms • Chatbots and Virtual Assistants • Voice Assistants • Retrieval Augmented Generation (RAG) solutions • Agentic AI applications  AI/LLM Evaluation Expertise Practical understanding of: • Prompt engineering principles • Context windows and memory handling • Function/Tool Calling • Retrieval pipelines • LLM evaluation methodologies • AI quality metrics and benchmarking Experience with: • Golden datasets • Human evaluation workflows • Statistical validation approaches • Non-deterministic output testing  Domain Knowledge Strong experience in Airline or Travel domain, including: • Flight Booking journeys • Reservation Management • Post-Booking Services • Check-In processes • Passenger servicing workflows  Preferred Skills • Experience with OpenAI, Azure OpenAI, Anthropic, Gemini, or similar AI platforms. • Understanding of NLP concepts and conversational design. • Exposure to API testing and automation tools. • Experience with CI/CD pipelines and test automation integration. • Knowledge of cloud platforms (Azure, AWS, GCP). • Familiarity with observability and monitoring tools for AI applications.