Business Data Insights Senior Specialist

Job description

We are looking for a skilled and versatile Data & AI Engineer who can own the full data-to-AI lifecycle — designing and building robust data pipelines using Azure Data Lake, Azure Data Factory, and Azure Databricks, and then using that data to build, fine-tune, and deploy machine learning and generative AI solutions with Azure Machine Learning and Azure OpenAI. The role requires strong hands-on experience across both disciplines: developing ETL/ELT pipelines, transforming structured and semi-structured data, and ensuring reliable data availability, as well as building RAG pipelines, integrating LLM-based solutions, and operationalizing models with MLOps best practices. The candidate should be comfortable working with business stakeholders, data scientists, BI teams, product teams, and technical teams to understand requirements, design scalable data and AI solutions, and deliver measurable business value end to end — from raw data to production-ready AI features. Job Description Role: Data & AI Engineer Work Experience 5+ years of combined experience in data engineering and AI/ML engineering, including building scalable data pipelines and cloud data platforms as well as developing and deploying machine learning or generative AI solutions, in an Agile or DevOps environment. Role Summary We are looking for a skilled and versatile Data & AI Engineer who can own the full data-to-AI lifecycle — designing and building robust data pipelines using Azure Data Lake, Azure Data Factory, and Azure Databricks, and then using that data to build, fine-tune, and deploy machine learning and generative AI solutions with Azure Machine Learning and Azure OpenAI. The role requires strong hands-on experience across both disciplines: developing ETL/ELT pipelines, transforming structured and semi-structured data, and ensuring reliable data availability, as well as building RAG pipelines, integrating LLM-based solutions, and operationalizing models with MLOps best practices. The candidate should be comfortable working with business stakeholders, data scientists, BI teams, product teams, and technical teams to understand requirements, design scalable data and AI solutions, and deliver measurable business value end to end — from raw data to production-ready AI features. Key Skills • 4+ years of experience spanning data engineering and AI/ML engineering, with strong exposure to data integration, ETL/ELT development, and cloud-based data and AI platforms. • Hands-on experience with Azure Data Factory for building, scheduling, monitoring, and managing data pipelines. • Working knowledge of Azure Data Lake Storage for storing, organizing, and managing large volumes of data. • Experience with Azure Databricks, including notebooks, Spark SQL, PySpark, data transformation, and performance optimization. • Strong SQL and Python skills for querying, transformation, data validation, troubleshooting, and performance tuning. • Hands-on experience with Azure Machine Learning or Azure AI Studio for training, deploying, and managing ML models. • Practical experience with Azure OpenAI Service or similar LLM platforms, including prompt engineering, fine-tuning, and model integration. • Experience building RAG pipelines using vector databases (e.g., Azure AI Search, FAISS, Pinecone) for grounding LLM responses in enterprise data. • Experience with ML/AI frameworks such as PyTorch, TensorFlow, scikit-learn, LangChain, or Semantic Kernel. • Good understanding of data lake architecture (bronze, silver, gold / medallion architecture) and MLOps practices (model versioning, CI/CD for ML, monitoring, retraining). • Experience with data modelling, data quality checks, data profiling, reconciliation, feature engineering, and model evaluation. • Exposure to DevOps practices such as Git, CI/CD, containerization (Docker, Kubernetes), version control, and deployment processes. • Working knowledge of Power BI or similar reporting tools, and knowledge of legacy ETL tools (Informatica, SSIS, Teradata, Oracle) will be an added advantage. Key Responsibilities • Design, develop, test, deploy, and maintain scalable data pipelines using Azure Data Factory, Azure Data Lake, Azure Databricks, SQL, and related technologies. • Build automated ETL/ELT pipelines to ingest, transform, validate, and publish data for analytics, reporting, and AI model consumption. • Design, build, and fine-tune machine learning and generative AI models to solve business problems, using the curated data pipelines as the foundation. • Develop RAG pipelines and integrate LLM-based solutions with enterprise data sources and applications. • Work with structured, semi-structured, and unstructured datasets, preparing them for both analytics and model training/evaluation. • Develop data and model transformation logic using SQL, PySpark, Spark SQL, Databricks notebooks, and Python ML frameworks. • Create and maintain reliable data flows and AI services across raw, curated, and consumption-ready layers, including production model inference. • Perform data exploration, validation, reconciliation, model evaluation, and performance testing to ensure accuracy and reliability end to end. • Collaborate with business stakeholders, data scientists, BI developers, product owners, and architects to convert requirements into technical data and AI solutions. • Support data migration, system integration, and AI feature rollout, from legacy platforms to cloud-based data and AI platforms. • Monitor pipeline and model performance in production, troubleshoot f