DataOps Technical Lead

Job description

Job Title: DataOps Technical Lead Databricks/Azure Work Location: Bangalore / Chennai / Gurgaon/ Pune/ Kolkata / Hyderabad Division/Department: Solution Support Employment Type: Full-time Job Description, Essential Duties and Responsibilities: • DataOps Lead is responsible for leading technical team and managing end-to-end project deliverables. • Should have strong experience and knowledge in DataOps support deliverables, KPIs and platform administration for Azure-Databricks DataOps project • Data Ops & Support roles • Pipeline Monitoring & Incident Response • Ticket/Incident Management • Data Anomaly Investigations & Root Cause Analysis • Hotfixes, Enhancements & Release Coordination • Creating Runbooks and Knowledge Base documents • Data Quality Tracking • Platform Administration • Databricks Platform Administration • Azure Infrastructure Management • Identity, Access & Governance • Data Pipeline Architecture & Platform Standards • Tools & Technologies • Databricks • Azure data services • SQL Server • Azure Synapse • Azure Data Factory (ADF) • Python and PySpark scriptingusing DLTs • Github • AI based tools (Claude, Co-Pilotetc) • Linear (task management) • ServiceNow[AG1] • Cosmos • ADO • Data pipelines: Monitor, identify issue, Debugging and Bug fixing of data pipelines. Should understand complex data pipelines and data engineering concepts. Mentoring and guiding junior team members. • Data Quality & Data Observability: Identify and fix the Data quality issues by performing RCA. Should have a good understanding of data observability tools like Unravel etc. • Data orchestration: Orchestration of jobs using Airflow. • DevOps: Should have a good understanding of the Dev Ops process. • Collaboration: Work closely with data engineering, data science, and IT teams to support data pipeline development, troubleshoot issues, and provide guidance on best practices. • Incident Management: Lead troubleshooting and resolution of issues related to Databricks clusters and jobs, ensuring minimal downtime and impact on business operations. • KPI: Responsible for managing MTTA, MTTR, IRR, response & resolution SLA and other key delivery matrix. • Upgrades & Patches: Manage upgrades, patches, and new releases of Databricks, ensuring minimal disruption to ongoing operations. • Documentation: Create and maintain detailed documentation of the Databricks environment, configurations, and procedures. • SOP: Able to build SOPs and set up DataOps best practices. • Continuous improvement and Innovation: Bringing continuous improvement and innovation for cost optimization, effort reduction, incidents reduction, platform stability and discuss with client, Development team for implementation. • Team Leading: Lead the technical team and manage day-to-day technical deliverables. • Escalation: Manage client’s day-to-day technical escalation • Stakeholder management: Able to handle key stakeholders, including client leadership. Skills, Education and Experience: • Having 8+ years of Azure-Databricks DataOps experience. • Team handling experience: Able to handle a team of 5-7 engineers • Open to work in UK & US Shift. • Working in agile methodology and Sprint framework • Must have education degree of B. Tech/B.E/MCA