Databricks - Principal Data Engineer

Job description

Req ID: 376003 NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Databricks - Principal Data Engineer to join our team in Bangalore, Karnātaka (IN-KA), India (IN). Experience: 10+ years overall | 5+ years hands-on DBT experience Function: Data & Analytics / Modern Data Engineering Role Type: Architecture, Consulting, CoE Leadership & Delivery Enablement About The Role We are looking for an experienced DBT Architect / DBT Center of Excellence (CoE) Lead to establish and lead our DBT capability and provide technical leadership across the organization's modern data engineering and analytics initiatives. The role will be responsible for building the DBT practice, standards, reference architectures, reusable accelerators, estimation frameworks, delivery methodologies, and engineering best practices while supporting global delivery teams across client engagements. The ideal candidate will combine deep hands-on expertise in DBT, cloud data platforms, modern data engineering, data architecture, and DevOps with strong consulting and stakeholder-management capabilities. This is a strategic and hands-on leadership role requiring the ability to work across pre-sales, solution architecture, delivery, capability development, modernization programs, and technology roadmaps . Key Responsibilities • DBT Center of Excellence Leadership • Establish and lead the DBT Center of Excellence across the Data & Analytics practice. • Define the DBT capability strategy, operating model, governance structure, and roadmap. • Establish enterprise-wide DBT development standards, coding guidelines, design patterns, naming conventions, testing standards, and deployment practices. • Define reusable reference architectures and implementation patterns for different DBT adoption scenarios. • Build a community of DBT architects, engineers, SMEs, and practitioners across delivery organizations. • Conduct technical forums, knowledge-sharing sessions, workshops, and DBT enablement programs. • Define competency frameworks and career paths for DBT practitioners. • Evaluate emerging DBT capabilities and recommend adoption strategies. • Solution Architecture & Technical Leadership • Architect scalable DBT-based modern data platforms and transformation frameworks. • Design DBT solutions across platforms such as Snowflake, Databricks, Redshift, BigQuery and other cloud data platforms. • Define DBT project structures, environments, deployment models, CI/CD architecture, orchestration, security, and governance. • Design solutions for: • Enterprise data warehouses • Lakehouse architectures • Data marts • Data products • ELT modernization • Legacy ETL modernization • Informatica/SSIS/Ab Initio and other ETL-to-DBT migrations • Define strategies for incremental processing, snapshots, SCD implementations, data quality, testing, lineage, observability, and performance optimization. • Provide technical governance and architecture reviews for major DBT implementations. • Troubleshoot complex performance, scalability, deployment, and architectural challenges. • Pre-Sales & Client Consulting • Support the sales organization in DBT-related pre-sales and client engagements. • Participate in client discussions, discovery workshops, technical presentations, demonstrations, and solutioning sessions. • Develop DBT solution proposals, architecture diagrams, migration approaches, and implementation strategies. • Lead technical responses for RFPs/RFIs/RFQs involving modern data engineering and DBT. • Identify opportunities to modernize legacy ETL platforms using DBT and cloud-native data platforms. • Develop solution narratives and differentiators for DBT-based offerings. • Support account teams in identifying and shaping new DBT opportunities. • Present technical solutions and roadmaps to senior client stakeholders. • Estimation & Delivery Planning • Define standardized DBT estimation frameworks and productivity benchmarks. • Develop estimation models based on: • Number of source systems • Number of ETL workflows • Transformation complexity • SQL complexity • Number of dependencies • Data volumes • Testing requirements • Migration patterns • Create effort estimation models for Simple / Medium / Complex DBT development and migration workloads. • Define team composition, skill requirements, productivity assumptions, and delivery capacity. • Support delivery teams in developing ROM, indicative, and detailed estimates. • Define delivery timelines and realistic productivity targets. • DBT Migration & Modernization • Define migration strategies for converting legacy ETL technologies into modern DBT architectures. • Establish migration assessment frameworks and factory-style delivery models. • Define migration waves, prioritization criteria, dependency management, and rollout strategies. • Develop approaches for automated or semi-automated migration of legacy ETL code. • Establish validation, reconciliation, parallel-run, and production cutover strategies. • Define approaches for migrating large-scale environments with thousands of ETL workflows and data objects. • Provide technical oversight for migration programs and resolve complex migration challenges. • Accelerators & Automation • Identify opportunities to build DBT accelerators, frameworks, utilities, and automation tools. • Lead development of accelerators for: • ETL-to-DBT conversion • SQL conversion • DBT model generation • Dependency analysis • Complexity assessment • Code quality assessment • Automated testing • Documentation generation • Lineage analysis • Migration assessment • Effort estimation • Deployment automation • DBT project scaffolding • Explore the use of Generative AI and Agentic AI for DBT development, migrat