AI Engineer

Job description

Role Overview We are seeking an AI Data Engineer who thrives at the intersection of Data Engineering and Autonomous AI. You will move beyond traditional ETL to build "AI-Ready" data pipelines and Agentic systems. Your role is two-fold: 1. CoE Accelerator Development: Architect and build internal frameworks and autonomous agents that automate complex data lifecycle tasks. 2. Client Delivery: Partner with clients to design and deploy sophisticated RAG (Retrieval-Augmented Generation) systems and Agentic workflows that can reason, plan, and execute data operations independently. Key Responsibilities ● Agentic Workflow Development: Design and deploy autonomous agents (using LangGraph, AutoGen, CrewAI) capable of orchestrating complex, multi-step data tasks, such as self-healing pipelines, automated data quality remediation, or autonomous SQL generation and execution. ● AI-Ready Data Pipelines: Architect robust pipelines using PySpark and Databricks to transform data into high-quality vectors and knowledge graphs optimized for Agentic memory and reasoning. ● Accelerators & Frameworks: Develop and maintain modular, reusable "Data Accelerators" that standardize Agentic orchestration, evaluation, and cost-monitoring for our CoE. ● Vector Database Management: Engineer, deploy, and manage vector indices (e.g., Databricks Vector Search, Pinecone) to serve as the long-term memory for AI agents. ● LLMOps & Monitoring: Implement observability frameworks to track agent performance, reasoning accuracy, and token costs. Integrate MLflow for experiment tracking. ● Strategic Collaboration: Act as a subject matter expert for the Data Practice CoE, contributing to technical whitepapers and the adoption of cutting-edge Agentic architectures. Technical Requirements ● Core Engineering: Expert-level proficiency in Python, PySpark, and SQL. ● Databricks Mastery: Hands-on expertise with the full Databricks ecosystem: Unity Catalog, Delta Live Tables (DLT), Workflows, and Serverless compute. ● Agentic & AI Orchestration: Strong experience building RAG pipelines and Agentic workflows using LangGraph, CrewAI, AutoGen, or LlamaIndex. This is the key differentiator for this role. ● Vectorization & Embeddings: Understanding of embedding models, chunking strategies, and the lifecycle of managing vector datasets for enterprise AI. ● Cloud Architecture: Familiarity with deploying AI-driven data solutions on AWS, Azure, or GCP. ● Tools & Methodologies: Experience with CI/CD (Git/GitHub Actions), containerization (Docker), and test-driven development. Preferred Qualifications ● Agentic Expertise (Huge Plus): Demonstrable experience in building autonomous agents that can troubleshoot, reason, or perform complex analytical tasks with minimal human intervention. ● Certifications: Databricks Certified Data Engineer Professional, Azure/AWS AI Engineer associate certifications. ● Full-Stack GenAI: Experience with frontend frameworks (Streamlit/Flask) to build rapid prototypes/PoCs of data accelerators. ● Governance: Familiarity with data security, PII masking, and access control models within an AI/Agentic context.