QA Automation Architect – AI/LLM Systems | 13+ years only | PAN India

Job description

**What impact will you make?** Every day, your work will make an impact that matters, while you thrive in a dynamic culture of inclusion, collaboration and high performance. As the undisputed leader in professional services **Deloitte is where you will find unrivaled opportunities to succeed and realize your full potential.** **The Team** Deloitte’s Technology & Transformation practice can help you uncover and unlock the value buried deep inside vast amounts of data. Our global network provides strategic guidance and implementation services to help companies manage data from disparate sources and convert it into accurate, actionable information that can support fact-driven decision-making and generate an insight-driven advantage. Our practice addresses the continuum of opportunities in business intelligence & visualization, data management, performance management and next-generation analytics and technologies, including big data, cloud, cognitive and machine learning. **QA Automation Architect – AI/LLM Systems** **Experience:** 13+ Years **Location:** PAN India **Employment Type:** Full-Time **About the Role** We are looking for a highly experienced **QA Automation Architect** to lead the quality engineering vision for **AI-driven and Generative AI applications** . This role goes beyond traditional QA—focusing on **LLM validation, AI safety, governance, and scalable automation architecture** , including integrations with enterprise platforms such as **Adobe Experience Manager (AEM)** . You will play a critical role in ensuring that AI systems are **accurate, reliable, safe, and compliant** , while driving enterprise-wide standards for **AI quality engineering** . **What You’ll Do** **Architect & Lead Quality Engineering** - Define and drive the **end-to-end QA strategy** for AI/LLM-based platforms - Design and implement **scalable automation frameworks** using **Playwright (TypeScript)** and **BDD (Python/Behave)** - Establish reusable **test architecture, patterns, and best practices** across teams - Drive **test strategies across UI, API, and CMS-driven applications (including AEM-based platforms)** - Champion **shift-left testing** and automation-first culture **AI/LLM Testing & Quality Validation** - Build frameworks to validate: - **Accuracy, relevance, and contextual correctness** - **Bias, fairness, and ethical AI behavior** - **Safety, toxicity, and harmful prompt handling** - **Fallback and edge-case scenarios** - Design and implement **LLM evaluation strategies** , including automated scoring and human-in-loop validation - Create **feedback loops and quality dashboards** for continuous improvement **Automation & API Engineering** - Architect robust **end-to-end and API test automation suites** - Drive high coverage across **UI, API, backend, and CMS layers (including AEM components and workflows)** - Implement **test data strategies** for deterministic and repeatable AI testing **Governance, Compliance & Observability** - Define and enforce **Responsible AI testing standards and governance frameworks** - Ensure **traceability, auditability, and regulatory compliance** - Establish standards for **prompt versioning, dataset control, and experiment tracking** - Validate **content workflows and publishing pipelines in systems like AEM** - Leverage **logs, traces, and observability tools** to validate system behavior **CI/CD & DevOps Integration** - Integrate automation into **CI/CD pipelines** with quality gates - Enable **test orchestration, parallel execution, and environment consistency** (Docker/Kubernetes) - Define and track **quality KPIs and SLAs** **Performance & Reliability** - Drive **performance testing** (p50/p95 latency, throughput, scalability) for AI systems - Validate performance across **APIs, AI services, and CMS platforms (e.g., AEM)** - Identify bottlenecks in **model inference and service layers** **Leadership & Influence** - Mentor QA engineers and SDETs; elevate team capabilities - Collaborate with **Engineering, Product, Data Science, CMS/AEM teams, and Security teams** - Act as a **thought leader in AI quality engineering and automation** **Must-Have Skills & Experience** - **13+ years** in QA/Automation with **architect-level experience** - Strong expertise in **Playwright + TypeScript (mandatory)** - Proven hands-on experience in **AI/LLM testing and validation** - Deep experience in building **scalable automation frameworks** - Strong experience with: - API testing ( **Playwright, Postman, RestAssured** ) - BDD ( **Python + Behave** ) - CI/CD tools ( **Jenkins, GitHub Actions, Azure DevOps, etc.** ) - Solid understanding of **Agile, DevOps, and modern engineering practices** **Good to Have** - Experience with **Generative AI systems in production** - Knowledge of **Responsible AI / AI Governance frameworks** - Exposure to **Cloud (AWS/Azure/GCP)** environments - Experience with **Docker, Kubernetes, microservices** - Understanding of **prompt engineering and versioning** - Experience working with **ML/Data Science teams** - Hands-on experience or exposure to **Adobe Experience Manager (AEM)** (component testing, workflows, content validation) - Background in **regulated domains** (Fintech, Healthcare, SaaS) **Why Join Us** - Work on **cutting-edge AI/LLM platforms** - Opportunity to **shape enterprise AI quality standards** - High-impact role with **architectural ownership and visibility** - Collaborative environment with **cross-functional innovation** **Ideal Candidate Profile** A seasoned QA leader who: - Thinks **architecturally, not just functionally** - Has a passion for **quality in AI systems and digital experience platforms (like AEM)** - Balances **technical depth with strategic vision** - Drives innovation while ensuring **safety and compliance** **How you will grow** At De