AI Solutions and Platforms Engineer

Job description

**Overview** Agentic AI Observability Senior Engineer is responsible for deploying, integrating, and operating a scaled Agentic AI observability platform across both internal and external agent frameworks. This role focuses on production-ready instrumentation and telemetry pipelines that provide end-to-end visibility across multi-step agent workflows—including planner/executor loops, tool/function calls, RAG retrieval, and memory/state—ensuring reliability, safety, performance, and cost governance at scale **Responsibilities** - Agentic AI Observability at Scale (0%) - Platform Deployment & Operations (Agentic AI Observability at Scale) - Deploy and run the Agentic AI observability platform across dev/uat/prod with HA, resiliency, and controlled rollouts - Implement release automation (CI/CD), canary deployments, rollback strategies, and configuration management for platform components - Own operational readiness: on-call runbooks, incident response, and production support for agent observability services - End-to-End Agent Workflow Tracing (Planner → Tools → Retrieval → Response) - Implement distributed tracing for full agent execution graphs, including correlation across: prompts, intermediate reasoning steps (where permitted), tool calls, external APIs, retrieval queries, and final responses - Enforce consistent trace context propagation, correlation IDs, and semantic conventions across agent services - Build instrumentation patterns to represent agent flows as spans (e.g., plan span, tool span, retrieval span, memory span, response span) - Agent Framework Integrations & Standardized Instrumentation - Deploy and maintain integrations for internal agent frameworks and external ecosystems such as Crew.ai, LangChain, Semantic Kernel, AutoGen, and custom orchestrators - Create reusable SDKs/middleware/sidecar patterns for teams to instrument agents with minimal effort - Define and implement tagging standards for: agent name/version, tool name, model provider, prompt template ID, retrieval source, tenant/app, and environment - Agentic AI Telemetry Pipelines & AI-Specific Signals - Build scalable pipelines for agent telemetry (logs/metrics/traces) using OpenTelemetry and platform observability tooling - Capture AI-specific metrics including: token usage, cost per task, tool-call latency, retrieval latency, grounding score proxies, error rates, and agent loop iterations - Implement sampling and redaction strategies for sensitive agent payloads (prompts, responses, retrieved content) aligned to governance requirements - Collaboration with Teams (10%) - Collaborate with transformation teams and business stakeholders to understand requirements and tailor AI agents to specific domains. - Work closely with AI platform teams to build scalable and cross-domain AI agents while ensuring end-to-end observability. - Integration & Deployment (10%) - Build and maintain CI/CD pipelines for agent services and operations center components, including automated testing and deployment - Automate onboarding for new agent use cases (templates, scaffolding, configuration checks) - Drive best practices for secure, scalable, and cost-effective agent deployments - Continuous Learning (10%) - Stay updated with the latest advancements in AI and machine learning technologies and integrate these into existing or new AI agents. - Conduct thorough testing and validation to ensure the reliability and accuracy of AI agents and solutions. **Qualifications** - Education: Bachelor’s or Masters in Computer Science, AI/ML, Data Science, or a related field. - Experience: 4-8+ years of software engineering experience; 2-3+ years building and observe AI/ML or GenAI applications preferred - Required Expertise: - Strong hands-on experience deploying observability solutions (Prometheus/Grafana/Elastic/Splunk/Datadog or equivalent) - Deep working knowledge of OpenTelemetry instrumentation and telemetry pipeline operations - Experience observing agentic AI systems: tool/function calls, orchestration, routing, memory/state, and RAG pipelines - Familiarity with Crew.ai, LangChain, Semantic Kernel, AutoGen, or similar agent frameworks - Experience with evaluation/quality monitoring and safe logging strategies for LLM systems - FinOps experience for tracking token and GPU spend, chargeback/showback, and cost anomaly detection - Experience implementing data governance controls for AI telemetry (PII redaction, retention, auditability) - Strong Kubernetes experience (AKS/EKS/GKE) including Helm, operators, ingress, and service networking - Strong automation skills (Python/Bash/Go) and CI/CD experience - Infrastructure-as-Code (Terraform/Bicep/CloudFormation) - Agent workflow tracing and telemetry correlation - Production operations and debugging distributed systems - Observability-as-a-platform enablement and automation - Strong documentation, collaboration, and stakeholder influence - Technical Proficiency: Implement monitoring for agent failure modes: tool-call failures, infinite loops, timeouts, hallucination risk signals, retrieval misses, and degraded response quality. Create alerts aligned to operational SLOs (availability, latency, tool reliability) and AI-specific indicators (cost spikes, loop bursts, retrieval anomalies). Support guardrail observability: policy blocks, content filtering events, and safety classifier outcomes (where applicable). Build onboarding automation (IaC, templates, CI checks) that makes observability “default-on” for all agentic services. - Problem-Solving: Ability to translate business challenges into technical solutions. - Collaboration Skills: Effective at working within cross-functional teams. - Agility: Flexibility to adapt to changing requirements and new technologies. - Communication Skills: Capable of explaining complex technical concepts to non-technical stakeholders.