SAP Datasphere & Databricks

Job description

Key Responsibilities: • Hybrid Data Architecture: Design and implement a seamless data mesh/lakehouse architecture integrating SAP Datasphere (Data & Business Layers and Data Spaces.) with the Databricks Lakehouse Platform. • Data Pipeline Engineering: Develop and maintain scalable, low-latency ETL/ELT pipelines using PySpark, Spark SQL, Delta Live Tables (DLT), and Datasphere Replication/Transformation flows. • Semantic Layer Integration: Expose SAP Datasphere analytical datasets and business models seamlessly to Databricks, ensuring business logic (like hierarchies and currencies) is preserved or effectively translated. • Federation vs. Ingestion Strategy: Establish governance on when to federate queries live from Databricks to Datasphere versus when to ingest and persist heavy SAP datasets into Delta Lake. • Unified Governance: Implement end-to-end data governance, lineage tracking and security matrix enforcement utilizing Databricks Unity Catalog alongside Datasphere Spaces and scoping rules. • Performance Optimization: Monitor and tune data transfer rates, cluster sizes, and query execution plans across both cloud platforms to minimize egress costs and maximize speed. Must Have Skills: Technical Core • SAP Datasphere Expertise: 2 years of production experience with SAP Datasphere, including Space Management, Data Builder, Business Builder, and integration with S/4HANA or BW/4HANA. • Databricks Mastery: 3+ years of hands-on experience building enterprise-grade lakehouses using Databricks, PySpark, and Delta Lake. • Advanced Data Ingestion: Deep understanding of modern integration patterns between SAP cloud products and hyperscaler storage (ADLS Gen2, AWS S3) via Kafka, SAP FedML, or OData/CDI streaming. • Data Governance: Strong experience with Unity Catalog for managing cross-platform data access and data lineage. • Cross-Ecosystem Fluency: Ability to speak the language of both traditional SAP functional teams (ABAP, S/4 schemas) and modern cloud data science teams. • Agile & DevOps: Experience with CI/CD tools (Git, Azure DevOps, GitHub Actions) adapted for Databricks notebooks and Datasphere transports. • Communication: Exceptional communication skills to present architectural decisions to executive stakeholders and technical teams alike.