Principal Software Engineer

Job description

At Cadence, we hire and develop leaders and innovators who want to make an impact on the world of technology. Senior AI Test / Automation Engineer Overview Role: AI Test / Automation Engineer Location: Bangalore India Department: AI Engineering / Quality Assurance Experience Level: Mid to Senior We are looking for a highly motivated Senior AI Test / Automation Engineer to design and scale automated validation frameworks for AI/ML models, LLM-based applications, and agentic systems. This role is critical to ensure that AI solutions meet enterprise standards for quality, reliability, safety, and compliance before and after production deployment. Key Responsibilities • Build and maintain AI test automation frameworks for pre-qualification and continuous validation of models and agent workflows • Develop comprehensive test suites, including: • Unit, integration, and end-to-end (E2E) • Functional, regression, performance, and safety testing • Validate AI system behavior, including: • Non-deterministic LLM outputs • Hallucinations and edge cases • Multi-step agent decision-making • Design and manage evaluation systems: • Golden datasets • Benchmarking pipelines (accuracy, latency, reliability) • Automate testing within CI/CD pipelines for model updates, prompt changes, and tool integrations • Implement observability and telemetry to enable traceability, monitoring, and audit readiness • Collaborate cross-functionally with ML, MLOps, Product, and Security teams to define quality gates and release criteria • Track and report quality KPIs, including test coverage, defect leakage, and system reliability • Drive root-cause analysis and continuous improvement across the AI testing lifecycle Required Skills Core Engineering • Strong programming skills in Python; familiarity with Bash, TypeScript, or Go • Experience with test automation frameworks such as PyTest, Playwright, Selenium, or Cypress • Proficiency in CI/CD tools (GitHub Actions, Jenkins, GitLab CI) • Experience with cloud platforms (AWS, Azure, GCP) and containers (Docker, Kubernetes) AI / ML & Agentic Systems • Hands-on experience with LLM ecosystems (OpenAI, Anthropic, Bedrock) • Familiarity with: • RAG architectures and vector databases (Pinecone, Weaviate) • Agent frameworks (LangChain, LlamaIndex, AutoGen) AI Testing Techniques • Experience with non-deterministic testing approaches (statistical assertions, tolerance thresholds) • Knowledge of evaluation methods: • LLM-as-a-judge • BLEU, ROUGE, semantic similarity scoring • Experience with prompt and agent regression testing • Understanding of AI safety testing, including adversarial testing, bias/fairness validation, and jailbreak detection Tooling (Preferred) • AI testing & observability tools: LangSmith, TruLens, Arize, Weights & Biases • Evaluation tools: DeepEval, Ragas, PromptFoo, Giskard • Monitoring: Prometheus, Grafana, OpenTelemetry Soft Skills • Strong analytical and problem-solving skills • Excellent communication and cross-functional collaboration • Data-driven mindset with focus on quality KPIs • Detail-oriented with a strong bias toward automation and scalability Experience Requirements • 7+ years in QA, SDET, or test automation engineering • Proven experience building and scaling automation frameworks • Hands-on experience with AI/ML systems or LLM-based applications • Experience testing RAG pipelines or agentic workflows • Owned end-to-end AI test strategy and architecture • Defined quality metrics and release gates • Delivered scalable validation pipelines for production AI systems • Supported audit and compliance readiness Preferred • Experience in enterprise or regulated environments (SOC2, ISO 27001, etc.) • Exposure to: • Shift-left testing practices • Production observability and monitoring • Chaos or resilience testing Senior-Level Differentiators Education • Bachelor’s or Master’s degree in Computer Science, Software Engineering, or related field Nice-to-have: • ISTQB certification • Cloud/ML certifications (AWS, Azure, GCP) • AI testing certifications What Success Looks Like • AI systems that are accurate, reliable, and safe • Fully automated test pipelines integrated into CI/CD • Measurable improvements in defect leakage and model quality • Strong observability and auditability across AI systems • Scalable validation frameworks supporting rapid AI innovation We’re doing work that matters. Help us solve what others can’t.