QA Automation Engineer - Enterprise Al & Agentic Systems

Job description

**Role Overview We are seeking a highly technical QA Automation Engineer with 4 to 5 years of experience to anchor the quality, reliability, and precision of our Enterprise Al Enablement initiative. In this role, you will architect the verification and evaluation layers for complex, multi-layered agentic systems and Retrieval-Augmented Generation (RAG) applications. You will lead the quality strategy for platforms that process complex unstructured documents and financial statements to surface strategic enterprise insights. You will ensure that Al primitives, agentic routers, and data pipelines are robust, explainable, and production-ready using a strict Test-Driven Development (TDD) mindset. Key Responsibilities 1. Agentic & RAG System Evaluation Design LLM Evaluation Metrics: Build automated evaluation suites to measure Al response quality, evaluating specifically for faithfulness, answer relevance, context recall, and hallucination reduction. Validate Agentic Routing: Code test scenarios to verify that Agentic Search Routers dynamically select the correct retrieval path (such as routing to a summary index versus a specific factual database query) based on user intent. Data Grounding Verification: Partner with development teams to validate that unstructured data ingested into semantic layers aligns accurately with baseline domain schemas. 2. Test-Driven Development (TDD) & Automation Frameworks Implement a TDD Approach: Author failing regression and integration tests before new Al primitives or data connector utilities are written, establishing a deterministic baseline for non-deterministic Al systems. Build End-to-End Test Harnesses: Standardize and maintain large automated test sets of queries and human-verified baseline responses. Al-Driven Automation: Leverage LLM-powered engineering utilities to accelerate test script generation, code coverage parsing, and synthetic data generation for boundary testing. 3. Pipeline Traceability & Cloud Performance Testing Pipeline Traceability Testing: Validate the integrity of cloud data pipelines as they scrape, ingest, and process unstructured data into central storage solutions. Provenance & Governance Verification: Audit governance gates, ensuring that confidence scoring, trust scoring, and provenance validations actively flag anomalies before responses reach end-users. Performance & Token Cost Optimization: Monitor and benchmark token utilization, execution latency, and API performance under heavy simulation loads to prevent cost explosions in production. Required Technical Skills & Qualifications Experience: 4 to 5 years of professional experience in QA Automation, with a focus on validating AI/ML pipelines, Large Language Model (LLM) applications, or complex distributed systems. Al Toolkits & Frameworks: Hands-on experience with LLM orchestration and evaluation libraries (e.g., LlamaIndex, LangChain). Al Evaluation Tooling: Direct experience with LLM observability, tracing, and evaluation platforms (e.g., BrainTrust, Ragas, TruLens, or LangSmith). Cloud & Protocol Architecture: Deep familiarity with cloud-native storage frameworks (such as AWS or Azure), cloud API management, and modern tool/data connection protocols (e.g., Model Context Protocol). Programming Proficiency: Strong coding skills in Python (for constructing index tests, manipulating data structures, and script automation) and SQL. Testing Paradigms: Expert-level knowledge of TDD/BDD practices, automated API testing, and backend testing frameworks (e.g., PyTest).**