Principal AI/ML Engineer

Job description

The Principal AI/ML Engineer is the technical leader responsible for designing, building, and scaling AI systems that combine LLM-powered GenAI and ADK-based agentic workflows on Google Cloud Platform. This role sets architecture standards, leads multi-team delivery, and governs safety, reliability, builds and manages the platform, and cost at enterprise scale—accelerating product teams to achieve 10× productivity through reusable patterns, platforms, and guardrails. **Key Responsibilities** **Strategy & Architecture** - Define reference architectures for GenAI apps, RAG systems, and agent ecosystems (single/multi-agent) on GCP using ADK. - Establish domain and platform standards: model selection, RAG/generation patterns, memory architectures, security baselines, observability, and LLMOps. - Lead portfolio-wide technical decisions (build/buy, vendor selection, SLAs, quotas) with a focus on reliability, safety, and cost control. **Solution Design & Delivery** - Architect and lead implementation of production-grade GenAI solutions (Vertex AI models, Grounding, Pipelines, Evaluation) and agentic services (planning, tools, memory, HIL). - Design multi-tenant and hub-and-spoke patterns with Okta/IAP/Apigee for secure API exposure and tenant isolation. - Drive end-to-end delivery across teams: data ingestion (Dataflow/Composer), indexing (BigQuery vectors/Vertex Vector Search), services (Cloud Run/Workflows), events (Pub/Sub). **Platformization & Reuse** - Build and maintain prompt libraries, tool catalogs, agent templates, and evaluation harnesses for organization-wide reuse. - Standardize LLMOps: CI/CD for prompts/models/agents, model registry, traceability, rollback, canaries, cost/performance scorecards. - Enable a marketplace of agents/services with productized APIs, documentation, chargeback, and KPIs. **Responsible AI, Security & Compliance** - Implement multi-layer guardrails: policy prompts, filters, memory governance, tool whitelisting, audit logs; ensure regulator-ready posture. - Codify privacy, PII handling, data residency, and per-tenant isolation using VPC-SC, Secret Manager, IAM, and Apigee policies. **Leadership & Enablement** - Mentor senior engineers and team leads; run architecture reviews, design clinics, and red-team exercises. - Drive continuous evaluation programs and publish org scorecards for quality, safety, and cost. - Partner with Product, Security, and SRE to align roadmaps, SLOs, and operational playbooks. **Required Technical Competencies** - LLM & GenAI: Model selection (Gemini & Model Garden), prompt engineering, RAG/grounding, multimodal pipelines, fine-tuning/adapter methods. - Agentic AI (ADK): Agent loops, planners, tool/function design, memory (episodic/semantic/long-term), HIL, policy enforcement. - Data & Retrieval: BigQuery (including vector functions), Vertex Vector Search, Document AI, Dataplex for lineage and governance. - Orchestration & Services: Cloud Run, Workflows, Pub/Sub, Dataflow/Composer; HA/DR, backpressure, circuit breakers. - LLMOps/MLOps: Vertex AI Pipelines, registry, CI/CD, trace correlation, cost/performance monitoring. - Security & Compliance: IAM, Secret Manager, VPC-SC, private service connect, DLP, Okta/IAP, Apigee API policies. - Observability & Cost: Central telemetry, user feedback loops, drift/outlier detection, quota/capacity planning. **Qualifications** - 12–15+ years in software/data/ML engineering; 2+ years hands-on with LLMs/GenAI and agentic systems. - Proven delivery of enterprise-scale GenAI/agent platforms on GCP (Vertex AI, BigQuery, Cloud Run, Pub/Sub, Workflows). - Demonstrated impact in platformization, governance, and multi-team technical leadership. - Strong proficiency in Python/TypeScript (or equivalent) and infrastructure-as-code (Terraform/GCP Deployment Manager). - Experience in security-by-design, privacy, and compliance audits. **Outcomes & KPIs (What “Great” Looks Like)** - Reliability: SLOs met (e.g., p95 latency, error budget adherence); audited HA/DR playbooks; zero Sev1 incidents due to preventable guardrail gaps. - Quality & Safety: Sustained improvements on faithfulness/toxicity/grounding scores; red-team findings resolved within agreed SLAs. - Cost & Performance: ≥ 30% reduction in run-cost via routing, caching, and prompt/template optimization; budget adherence per tenant. - Productivity & Reuse: ≥ 50% reuse of tools/templates across teams; time-to-market reduced by ~40% for new AI features. - Adoption & Enablement: ≥ 3 cross-domain AI capabilities launched per quarter; engineers enabled through patterns and training. **Core Responsibilities (Day-to-Day)** - Own reference architectures and standards for GenAI and Agentic AI on GCP. - Lead design reviews and production readiness assessments. - Curate and evolve prompt/agent/tool libraries with versioning and documentation. - Establish evaluation harnesses (golden sets, scenario tests, trace replay, chaos for agents). - Partner with SRE/Platform to implement observability, alerts, feature flags, canaries, and rollback mechanisms. - Drive security reviews, policy-as-code, and auditability for all AI systems. **Demonstrated Behaviors (Principal Level)** **Technical Leadership** - Systems thinking: Anticipates failure modes, cost implications, and long-term maintenance; makes reversible vs. irreversible decision trade-offs explicit. - Pragmatic innovation: Balances cutting-edge methods (e.g., learned planners, multimodal grounding) with operational simplicity and reliability. - Platform-first mindset: Designs for reuse; evangelizes patterns; prevents bespoke one-offs unless clearly justified. **Execution Excellence** - Outcome orientation: Frames problems with clear KPIs; selects the simplest architecture that satisfies reliability, safety, and cost. - Bias to automation: Converts manual steps into workflows, CI/CD pipelines, and platform capabilities; eliminates toil. - Operational rigor: Treats prompts/models/agents as ve