Quality Assurance Tester

Job description

**QA Tester ( AI Observability & Monitoring)** **Role Overview** We are seeking **a QA Tester specializing in AI Observability and Monitori** ng to support the validation and continuous monitoring of AI/ML solutions in a regulated enterprise environment. This role will focus on ensuring that AI systems ar **e traceable, explainable, and continuously performing as expect** ed by validating observability frameworks, monitoring pipelines, and model performance metrics. The candidate will work closely with AI engineers, data scientists, and validation teams to ensure AI solutions meet quality, compliance, and audit readiness standard **Key Responsibilities** **1. AI Observability Validation** - Validate observability instrumentation across AI systems, including - Input/output tracing Telemetry data (latency, token usage, cost, - Ensure all observability signals are captured, linked, and auditable - Verify traceability and explainability of model behavior across workf **lows** **2. AI Model Monitoring & Drift Te** sting - Validate monitoring frameworks for Model performance (accuracy, confidence, consistency) - Drift detection and threshold-based Testing mechanisms and escalation workflows for Performance degradation Anomalous outputs - Support continuous monitoring validation in production environ **ments** **3. AI Behavior & Functional T** esting Design and execute test scenarios covering Edge cases and ambiguous inputs Prompt variations and response consistency Bias and fairness validation Validate model outputs against expected results and SME benchmarks Perform comparative validation (AI vs. baseline/manual ou **tputs)** **4. Observability Tools & Integration** Testing Test integration between AI applications and observability tools (e.g., Langfuse or similar platforms) Validate data pipelines feeding observability dashboards and KPI metrics Ensure end-to-end visibility across AI lifecycle (development B'\\*(J QA B'\\*(J prod **uction)** **5. Non-Functional & System Quality** - Testing Validate non-functional requirements including Performance and latency Reliability and resilience Logging and auditability - Ensure monitoring coverage aligns with enterprise quality and governance S **tandards** **6. Audit, Compliance & Docu** mentation Maintain audit-ready documentation for Test cases, execution results, and validation evidence Ensure alignment with SDLC validation processes AI governance and compliance requirements Support inspection readiness and audit responses **as needed** **Required Qualification** - Education Bachelors degree in Computer Science, Data Science, Engineering, or re **lated fiel** d - Experience 3 to 7 years of experience in QA / Testing/ Validation Experience w **orking with AI/ML systems or data-driven** applications - **Exposure to monitoring systems, logging frameworks, or observability platforms** **Tec** hnical Skills Strong understanding of AI/ML concepts (LLMs, model behavior, drift, evaluation metrics) Experience with API testing and backend validation SQL / data validation techniques Familiarity with Observability tools (e.g., Langfuse, logging/monitoring platforms)Test management tools (e.g., QTes **t, ALM tools)** **QA & Val** idation Skills Experience designing: Functional and non-functional test scenarios Edge case and negative testing scenarios Understanding of Test automation concepts (Python preferred) End-to-end valida **tion lifecycle and** analytical Skills Ability to Interpret AI model outputs and identify anomalies Analyze trends and detect inconsistencies in **model behavior** **Preferre** d Qualificatio **ns** **Experience in Gen** AI / LLM testing Knowledge of Prompt engineering and evaluation methods Familiarity with GxP / regulated industry environments AI governance, explainability, and Responsible AI frameworks