Data Engineer

Job description

**Job Purpose** To design and deliver AI-powered BI solutions by integrating Databricks Genie, Semantic Models, and Metric Views on top of a robust end-to-end data engineering stack. The role focuses on enabling self-service analytics using natural language (GenAI), ensuring business-friendly data consumption, and building governed, scalable, and high-performance data ecosystems across batch and real-time pipelines. **Duties and Responsibilities** KEY ROLES / PRINCIPAL ACCOUNTABILITIES **AI for BI GenAI Enablement** - Design and implement GenAI-powered BI solutions using Databricks Genie or MS Fabric - Create and manage Data Rooms for business users with curated datasets - Define and maintain Instructions Layer (prompt engineering for business context) - Enable natural language to SQL/insights workflows for self-service analytics - Drive adoption of Databricks One other platforms as a unified analytics interface **Semantic Layer Metrics Engineering** - Design and manage Semantic Models for business abstraction - Develop reusable and governed Metric Views (KPIs, aggregations, business definitions) - Ensure consistency across BI tools (Power BI / Genie) - Align semantic layer with business glossary and data governance policies **Data Engineering Platform Development** - Build scalable pipelines using Azure Databricks (PySpark, SQL) - Develop and orchestrate ETL workflows using Azure Data Factory - Work with Delta Lake architecture (BronzeSilverGold layers) - Enable real-time and batch data processing pipelines - Enable data exposure via APIs for BI and downstream systems **CI/CD DevOps** - Implement CI/CD pipelines for data and AI workflows - Automate deployments across environments (Dev, QA, Prod) - Ensure version control and reproducibility **KEY RESPONSIBILITIES** - Translate business problems into AI-driven BI solutions - Own end-to-end delivery: ingestion transformation semantic layer AI consumption - Design scalable data architecture aligned with lakehouse principles - Ensure data quality, governance, and metric consistency - Collaborate with business teams to onboard them onto Genie-based analytics - Optimize performance of queries, pipelines, and AI responses - Establish best practices for AI in BI (prompting, semantic tuning, governance) - Drive adoption of self-service BI with minimal dependency on tech teams **Key Decisions / Dimensions** - Define semantic layer design and metric definitions - Decide GenAI prompting strategies and instruction frameworks - Prioritize data vs AI optimization trade-offs - Handle production issues with RCA and long-term fixes - Drive architectural decisions for lakehouse + AI integration **Major Challenges** - Ensuring accuracy and trust in AI-generated insights - Driving adoption of GenAI-based BI over traditional dashboards - Maintaining semantic consistency across multiple tools - Balancing performance, cost, and scalability - Managing dependencies across data engineering, AI, and business teams **Required Qualifications and Experience** REQUIRED SKILLS EXPERIENCE Must Have - Azure Databricks PySpark, SQL, Delta Lake - Strong experience in Semantic Modeling Metrics Layer design - Hands-on with Databricks Genie / GenAI-based BI workflows - Databricks One (Unified BI Experience) - Prompt Engineering / Instruction tuning for AI systems - Python (Pandas, PySpark, FastAPI) - Azure Data Factory (ADF) for ETL pipelines - Strong SQL and data modeling skills Good to Have - Cosmos DB / MongoDB (NoSQL concepts) - Azure Data Explorer (KQL) DATA STACK (MANDATORY FOR SCREENING) SNoData Platform / ConceptsAssociated Technologies 1. Databricks Lakehouse - PySpark, SQL, Delta Lake 2. AI for BI - Databricks Genie, Genie Rooms, Instructions, Agents 3. Semantic Layer - Semantic Models, Metric Views 4. ETL Orchestration - Azure Data Factory 5. Programming - Python, C#/.NET 6. Cloud Platform - Azure (Preferred) 7. DevOps - CI/CD Pipelines, Git