DataOps Technical Lead

Job description

**Job Title: DataOps Technical Lead Databricks/Azure** **Work Location:** Bangalore / Chennai / Gurgaon/ Pune/ Kolkata / Hyderabad **Division/Department:** Solution Support **Employment Type: Full-time** **Job Description, Essential Duties and Responsibilities:** - **DataOps Lead is responsible for l**eading technical team and managing end-to-end project deliverables. - Should have strong experience and knowledge in DataOps support deliverables, KPIs and platform administration for Azure-Databricks DataOps project - **Data Ops & Support roles** - Pipeline Monitoring & Incident Response - Ticket/Incident Management - Data Anomaly Investigations & Root Cause Analysis - Hotfixes, Enhancements & Release Coordination - Creating Runbooks and Knowledge Base documents - Data Quality Tracking - **Platform Administration** - Databricks Platform Administration - Azure Infrastructure Management - Identity, Access & Governance - Data Pipeline Architecture & Platform Standards - **Tools & Technologies** - Databricks - Azure data services - SQL Server - Azure Synapse - Azure Data Factory (ADF) - Python and PySpark scriptingusing DLTs - Github - AI based tools (Claude, Co-Pilotetc) - Linear (task management) - ServiceNow[AG1] - Cosmos - ADO - **Data pipelines:** Monitor, identify issue, Debugging and Bug fixing of data pipelines. Should understand complex data pipelines and data engineering concepts. Mentoring and guiding junior team members. - **Data Quality & Data Observability:** Identify and fix the Data quality issues by performing RCA. Should have a good understanding of data observability tools like Unravel etc. - **Data orchestration:** Orchestration of jobs using Airflow. - **DevOps:** Should have a good understanding of the Dev Ops process. - **Collaboration:** Work closely with data engineering, data science, and IT teams to support data pipeline development, troubleshoot issues, and provide guidance on best practices. - **Incident Management:** Lead troubleshooting and resolution of issues related to Databricks clusters and jobs, ensuring minimal downtime and impact on business operations. - **KPI:** Responsible for managing MTTA, MTTR, IRR, response & resolution SLA and other key delivery matrix. - Upgrades & Patches: Manage upgrades, patches, and new releases of Databricks, ensuring minimal disruption to ongoing operations. - **Documentation:** Create and maintain detailed documentation of the Databricks environment, configurations, and procedures. - **SOP:** Able to build SOPs and set up DataOps best practices. - **Continuous improvement and Innovation:** Bringing continuous improvement and innovation for cost optimization, effort reduction, incidents reduction, platform stability and discuss with client, Development team for implementation. - **Team Leading:** Lead the technical team and manage day-to-day technical deliverables. - **Escalation:** Manage client’s day-to-day technical escalation - **Stakeholder management:** Able to handle key stakeholders, including client leadership. **Skills, Education and Experience:** - Having 8+ years of Azure-Databricks DataOps experience. - Team handling experience: Able to handle a team of 5-7 engineers - Open to work in UK & US Shift. - Working in agile methodology and Sprint framework - Must have education degree of B. Tech/B.E/MCA