SENIOR DATA ENGINEER - Azure Data Factory

Job description

Job Description?? Job Title:?Data Engineer Location:?Bengaluru (In Office)? 1. Purpose of the role? ?AB InBev?is?seeking a skilled Azure Data Engineer with a minimum of 4 years of experience to join our dynamic team. The ideal candidate?should?have a strong background in Azure Data Factory (ADF), Databricks, and Delta tables, along with?proficiency?in?PySpark?and SQL. Experience with Feature Store is considered a significant advantage.??? 1. Key tasks & accountabilities?? • Responsible for Creation of ETL Pipelines.? • Responsible for designing and building new Data Models and?optimizing?existing data models.? • Exploit elasticity and agility of the cloud.? • Ownership of physical Data models of serving layer? • Responsible and accountable for pipelines reliability and availability.? • Responsible of Data analysis for troubleshooting and exploration? • Understand the meaning,?domain?and the context of the data.? • Professional & Technical skills? 1. Essential:?? • Azure Technology Stack: SQL DW/Synapse- Azure Data Factory - Azure Databricks - Azure Data Lake Storage.? • Ability to write complex SQL scripts and stored procedures.? • Demonstrated experience in design and delivering data platforms for Business Intelligence or Data Warehouse, including data ingestion, ETL and data integration.? • Strong skills in handling and analysing complex, high volume data with excellent attention in details.? • Familiar with Agile?methodology?and Agile working environment.? • Ability to work along with team leads,?BAs, data architects, other.? • Develop, design, and implement data pipelines using Azure Data Factory for efficient data ingestion, transformation, and processing.? • Utilize Databricks to build scalable and optimized data processing workflows and perform advanced analytics tasks.? • Design and manage Delta tables for handling large-scale?structured and semi-structured data efficiently.? • Collaborate with cross-functional teams to understand data requirements and translate them into technical solutions.? • Optimize?and fine-tune?PySpark?scripts for performance improvement and scalability.? • Create and?maintain?SQL queries for data extraction, manipulation, and analysis.? • Leverage Feature Store for efficient management and sharing of features across machine learning pipelines (added advantage).? • Monitor and troubleshoot data pipeline issues to ensure data integrity and reliability.? • Stay updated with the latest Azure data engineering technologies and best practices.? • Leverage Git, manage Release Cycles.? 1. Qualifications, Experience, Skills? • Bachelor's degree in Computer Science, Information Technology, or related field.? • Minimum of 4 years of professional experience as a Data Engineer, with a focus on Azure technologies.? • Proficiency?in Azure Data Factory for building and orchestrating data pipelines.? • Strong experience with Databricks for data processing, analytics, and machine learning tasks.? • Solid understanding of Delta tables and their usage for managing large-scale data sets.? • Expertise?in?PySpark?for data manipulation, transformation, and analysis.? • Proficient in SQL with hands-on experience in writing complex queries.? • Experience with?Feature?Store is a significant advantage.? • Strong analytical and problem-solving skills with attention to detail.? • Excellent communication and collaboration skills to work effectively in a team environment.? • Azure certification(s) such as Azure Data Engineer Associate is a plus.? 1. Desirable:? • Experience in working on AI and ML related projects to prepare the necessary data and data models.