QA Engineer

Job description

Key Responsibilities • Design and execute test strategies specifically for AI/ML models, LLM-based applications, and data pipelines. • Develop automated test frameworks for model validation, regression testing, and performance benchmarking. • Evaluate model outputs for accuracy, consistency, relevance, hallucination, and bias across diverse inputs. • Test RAG (Retrieval-Augmented Generation) pipelines, chatbots, recommendation systems, and other AI- driven features. • Collaborate with data scientists and ML engineers to define acceptance criteria and quality thresholds. • Build and maintain evaluation datasets, ground truth sets, and adversarial test cases. • Monitor models in production for drift, degradation, and anomalous behavior. • Validate data quality, data pipelines, and feature stores that feed AI systems. • Document defects, edge cases, and failure patterns specific to AI behavior. • Ensure AI systems meet ethical, fairness, and compliance standards (bias audits, explainability checks). Required Experience • 3–6 years of overall QA experience, with at least 1–2 years specialized in AI/ML quality assurance. • Bachelor's or Master's degree in Computer Science, Engineering, or a related field. • Strong proficiency in Python for test automation and data analysis. • Familiarity with LLM evaluation frameworks (e.g., RAGAS, DeepEval, Promptfoo, LangSmith). • Hands-on experience with testing tools: Pytest, Selenium, Postman, or similar. • Solid understanding of the ML lifecycle — training, validation, deployment, and production monitoring. • Knowledge of data quality tools and pipeline testing (e.g., Great Expectations, dbt tests). • Analytical and inquisitive mindset with red-team thinking to challenge model outputs effectively. • Strong documentation, communication, and cross-functional collaboration skills. Preferred Skills • Experience with prompt engineering and red-teaming LLMs. • Familiarity with MLOps platforms (e.g., MLflow, SageMaker, Vertex AI). • Knowledge of vector databases and embedding quality evaluation. • Understanding of AI safety, responsible AI principles, and fairness frameworks. • Experience with A/B testing and shadow deployment strategies for AI models. Pay: From ₹1,000,000.00 per year Work Location: In person