AI Quality & Evaluation Engineers

Job description

**Essential Functions** - Develop and execute prompt testing strategies, including robustness, sensitivity, and ambiguity handling - Design and implement evaluation frameworks for LLMs and agent-based systems, including metrics for accuracy, consistency, and reliability - Validate AI outputs through data cross-checking, failure mode analysis, and edge case testing - Test agentic workflows end-to-end, ensuring correct decision-making, orchestration, and failure handling - Build and maintain automated testing and benchmarking pipelines, integrated into CI/CD - Investigate data quality issues and output inconsistencies, and define validation strategies - Contribute to Python-based tooling supporting evaluation, testing, and data validation - Collaborate with cross-functional teams to improve AI quality, reliability, and auditability. **Qualifications Required** - BSc/MSc in Computer Science, Data Science, Mathematics, Physics, or a related field - Strong Python skills, particularly for Data processing, test automation, evaluation pipelines - Experience with APIs, services, data pipelines, integration or system level testing - At least 6 months of hands-on experience with AI-enabled systems(e.g., LLM-based applications, generative AI features, or data-driven systems) - Strong analytical and problem-solving skills, with the ability to Evaluate outputs critically and to Identify patterns and failure modes.