SAP Datasphere & Databricks

Job description

**Key Responsibilities:** - **Hybrid Data Architecture:** Design and implement a seamless data mesh/lakehouse architecture integrating SAP Datasphere (Data & Business Layers and Data Spaces.) with the Databricks Lakehouse Platform. - **Data Pipeline Engineering:** Develop and maintain scalable, low-latency ETL/ELT pipelines using PySpark, Spark SQL, Delta Live Tables (DLT), and Datasphere Replication/Transformation flows. - **Semantic Layer Integration:** Expose SAP Datasphere analytical datasets and business models seamlessly to Databricks, ensuring business logic (like hierarchies and currencies) is preserved or effectively translated. - **Federation vs. Ingestion Strategy:** Establish governance on when to federate queries live from Databricks to Datasphere versus when to ingest and persist heavy SAP datasets into Delta Lake. - **Unified Governance:** Implement end-to-end data governance, lineage tracking and security matrix enforcement utilizing **Databricks Unity Catalog** alongside Datasphere Spaces and scoping rules. - **Performance Optimization:** Monitor and tune data transfer rates, cluster sizes, and query execution plans across both cloud platforms to minimize egress costs and maximize speed. **Must Have Skills:** **Technical Core** - **SAP Datasphere Expertise:** 2 years of production experience with SAP Datasphere, including Space Management, Data Builder, Business Builder, and integration with S/4HANA or BW/4HANA. - **Databricks Mastery:** 3+ years of hands-on experience building enterprise-grade lakehouses using Databricks, PySpark, and Delta Lake. - **Advanced Data Ingestion:** Deep understanding of modern integration patterns between SAP cloud products and hyperscaler storage (ADLS Gen2, AWS S3) via Kafka, SAP FedML, or OData/CDI streaming. - **Data Governance:** Strong experience with **Unity Catalog** for managing cross-platform data access and data lineage. - **Cross-Ecosystem Fluency:** Ability to speak the language of both traditional SAP functional teams (ABAP, S/4 schemas) and modern cloud data science teams. - **Agile & DevOps:** Experience with CI/CD tools (Git, Azure DevOps, GitHub Actions) adapted for Databricks notebooks and Datasphere transports. - **Communication:** Exceptional communication skills to present architectural decisions to executive stakeholders and technical teams alike.