Distinguished Engineer, Data Platforms

Job description

About the Role: Grade Level (for internal use): 13Key Responsibilities Data Pipeline Transition and Platform Delivery • Partner with the core Databricks team to plan and execute the transition of existing data pipelines to the target enterprise data platform. • Drive implementation of repeatable engineering patterns for ingestion, transformation, testing, deployment, and monitoring across onboarded datasets. • Guide the design and operation of cloud-native data pipelines leveraging relevant AWS services such as: • Amazon S3 for durable storage • AWS Glue for integration and catalog-driven processing • AWS Lambda for event-driven processing • Amazon Kinesis for streaming use cases • AWS Lake Formation for governed data lake controls • Promote the use of AWS IAM, encryption, and environment-level controls to enforce secure access to platform resources and data products in line with enterprise governance expectations. • Influence architectural decisions related to Databricks, Delta Lake, Apache Iceberg, Unity Catalog, metadata-driven processing, and governed data lake design. AI-Assisted Engineering and Context Engineering • Champion the adoption of AI-powered development tools, including Claude, GitHub CoPilot, LLMs, and other AI-assisted engineering solutions, to increase engineering velocity, code quality, and operational effectiveness. • Apply LLM-based tools to support common data engineering activities such as: • Code generation • Code refactoring • SQL optimization • PySpark optimization, etc. • Develop and implement context engineering practices that provide AI tools with the necessary technical background, architectural constraints, data schemas, coding standards, platform patterns, security requirements, and governance expectations. • Create reusable context packs, prompt libraries, and AI-enabled engineering playbooks for common data platform tasks, including Databricks pipeline migration, ingestion framework development, test generation, code review support, and operational troubleshooting. • Use LLMs to accelerate understanding and modernization of legacy pipelines, including dependency analysis, code explanation, transformation logic interpretation, metadata extraction, and refactoring recommendations. Data Onboarding and Asset-Agnostic Enablement • Provide technical direction for offshore teams supporting data onboarding to the enterprise platform in an asset-agnostic manner. • Define and operationalize onboarding patterns that can support a broad range of data assets, domains, and source systems without requiring bespoke platform redesign for each use case. • Work with partner teams to simplify and standardize how data is ingested, transformed, governed, and published to the platform. • Establish reusable ingestion and processing patterns across batch and streaming use cases using technologies such as AWS Glue, AWS Lambda, Amazon Kinesis, or event-driven integrations where appropriate. Data Mastering Platform Integration • Support integration of platform pipelines and datasets with the enterprise data mastering platform. • Collaborate with upstream and downstream stakeholders to ensure mastered data can be consumed reliably through standardized interfaces and governed data flows. • Help establish data quality controls, reconciliation processes, metadata alignment, and stewardship workflows required to support trusted mastered data in the platform. • Contribute to issue resolution and continuous improvement related to mastering-related ingestion and distribution workflows. • Support data mastering capabilities aligned with platforms such as NeoXam DataHub, including: Data acquisition, Cleansing, Enrichment, Mastering, Reconciliation, Golden copy generation & Downstream distribution of trusted data products. Semantic Modeling and Data Discoverability • Support semantic modeling practices that help create consistent business definitions, reusable data concepts, and governed consumption patterns across the enterprise data platform. • Partner with business, architecture, data governance, and platform teams to align technical data structures with business-friendly semantic definitions. • Contribute to defining common entities, attributes, relationships, hierarchies, metrics, and business terms across key data domains such as instruments, issuers, accounts, portfolios, risk, market data, reference data, and investment data. • Help connect semantic modeling concepts with metadata management, business glossaries, data catalogs, lineage, and governed data products. Technical Influence and Engineering Excellence • Serve as a senior technical expert for offshore data platform engineering, providing guidance on architecture, design patterns, implementation quality, and production readiness. • Influence architectural decisions across pipeline migration, lakehouse design, cloud-native data engineering, metadata-driven processing, AI-assisted engineering, and governed data platform operations. • Provide technical guidance to engineers and partner teams without direct people management responsibility. • Contribute to design reviews, code reviews, implementation planning, and technical problem-solving for complex data platform initiativ