Azure Databricks Consultant

Job description

## **Job Description:** - Infosys is seeking a highly skilled Azure Databricks Engineer with 6 to 8 years of experience in designing developing and optimizing cloud based data engineering solutions - The ideal candidate will possess strong expertise in Azure Databricks PySpark Azure Data Factory ADF Azure Data Lake Storage ADLS SQL and Data Engineering practices ## **Key Responsibilities:** - Design develop and maintain scalable Azure Databricks solutions for enterprise data platforms - Build and optimize end to end ETL ELT pipelines using PySpark Spark SQL and Azure Data Factory ADF - Develop robust data ingestion frameworks from multiple structured and unstructured data sources - Implement and manage Delta Lake and Lakehouse Architecture for high volume data processing - Create and optimize data models transformations and workflows to support analytical and business requirements - Perform performance tuning and troubleshooting of Databricks workloads and Apache Spark jobs - Work with Azure services including ADLS Gen2 Azure Synapse Analytics Azure Key Vault and Azure Data Factory - Collaborate with business stakeholders architects and cross functional teams to deliver data solutions - Ensure adherence to data governance security and best practices - Participate in code reviews CI CD implementation testing deployment and production support activities ## **Technical Requirements:** - 6 to 8 years of experience in Data Engineering and Big Data technologies - Strong hands on experience with Azure Databricks and PySpark - Expertise in Apache Spark Spark SQL and distributed data processing - Experience with Azure Data Factory ADF and building enterprise scale data pipelines - Strong knowledge of Azure Data Lake Storage ADLS Gen2 ## **Preferred Skills:** Technology->Cloud Integration->Azure Data Factory (ADF),Technology->Cloud Platform->Azure Core services->Azure Virtual Machines,Technology->Cloud Platform->Azure Networking Services->Azure Application Gateway,Technology->Data Engineering->Databricks