Senior Data Architect – AWS & Databricks Modernization

Job description

At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work—but it’s work worth doing. If you’re driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us. CTI MD Tech@Lilly Senior Data Architect — AWS & Databricks Modernization at Scale Position Description About Lilly At Lilly, everything we do starts with patients. We unite caring with discovery to make life better for people around the world. Headquartered in Indianapolis, Indiana, our global team of over 50,000 employees work with urgency and purpose to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. We bring our best to this work because people depend on it. If you're driven by purpose and determined to make a meaningful difference for patients, we invite you to bring your skill and your commitment to Lilly. About Technology@Lilly At Lilly, technology is not a support function. It is how a global medicine company operates, innovates, and delivers. Lilly in Bengaluru builds the capabilities that make this possible, cloud platforms, AI systems, and automation at enterprise scale, all in service of a purpose that makes this technology work genuinely distinctive, from advancing drug discovery to enabling connected clinical trials to keeping a global medicine company running at the standard patients deserve. About the Business Function: The Clinical & Non-Clinical Data Organization at Eli Lilly and Company is responsible for the design, build, and operation of enterprise data platforms that power drug discovery, clinical development, and regulatory submissions. Data Hub is building a robust Data Strategy to make Lilly's Clinical and Non-Clinical data AI-ready and audit-ready, delivering scalable, governed, and reusable data products that accelerate how medicines reach patients. Our data engineering organization sits at the intersection of science, technology, and patient impact — connecting Clinical and Non-Clinical data across the full chain, from ingestion to consumption. Role: Senior Data Architect — AWS + Databricks Modernization at Scale The Senior Data Architect (R5) is a hands-on Databricks and lakehouse modernization leader who owns the architecture and execution of large-scale migrations from legacy warehouses and point platforms onto a unified Databricks lakehouse for the Clinical and Non-Clinical data domain. This is a builder role: the architect writes code, builds migration pipelines, and personally ships production lakehouse assets — in addition to defining the modernization roadmap and rollout patterns that scale across dozens of domains. The role also owns the roadmap for semantic modeling and analytics-ready data, builds reusable data products the organization can adopt, drives data quality on clinical data specifically, and defines the metrics that enable data-driven decision making. AI-assisted and agentic ways of working are the expected default. Domain knowledge of the clinical data landscape is an added advantage. This role is split 70% hands-on technical execution including coding and 30% strategy, and shapes the future technology landscape. Key Responsibilities: Lakehouse Modernization & Migration at Scale (Hands-On) • Own the technical roadmap and execution for migrating legacy warehouses, on-prem databases, and point data platforms onto Databricks — sequencing dozens of Clinical and Non-Clinical domains with minimal disruption. • Personally build migration pipelines and re-platforming accelerators (schema conversion, historical backfill, dual-run validation, cutover automation) that move data at scale with parity and auditability. • Define reusable modernization patterns — landing zone design, medallion (bronze/silver/gold) conventions, workspace/catalog topology — that scale consistently as new domains onboard. • Right-size and standardize the Databricks platform footprint across environments (dev/test/prod, multiple workspaces) for cost, performance, and governance at enterprise scale. • Establish cutover, rollback, and data-reconciliation practices that de-risk large-scale migrations in a regulated environment. Databricks Lakehouse Architecture & Engineering (Hands-On) • Architect and build the Clinical/Non-Clinical lakehouse on Databricks, applying medallion design across Delta Lake tables at scale across multiple domains. • Personally build and optimize Delta Live Tables (DLT) pipelines, Databricks Workflows, and PySpark/Spark SQL jobs for high-volume ingestion, transformation, and curation. • Own Unity Catalog design and rollout — catalogs, schemas, access control, lineage, and data sharing — as the governance backbone across an expanding domain footprint. • Tune performance and cost at scale: Photon, cluster policies, job/task orchestration, auto-scaling, and Databricks SQL Serverless warehouses across many concurrent workloads. • Package and deploy pipelines using Databricks Asset Bundles and CI/CD; evaluate Lakehouse Federation and cross-workspace patterns for enterprise-wide access. AI-Native & Agentic Data Engineering • Use AI-assisted and agentic tooling by default — migration gap analysis, pipeline scaffolding, code conversion, and documentation — to accelerate modernization at scale. • Apply Databricks Mosaic AI / MLflow and LLM-based tooling to automate schema mapping, ontology alignment, and data-quality scoring during migration. • Build reusable AI-assist