Data Engineer

Job description

**Azure data bricks with fact \& dimension table, Spark SQL, Pyspark, autoloader, unity catalogue \& file format JSon - Mandatory** **Mandate Skills : Hands on experience in Azure Databricks \& Medallion Architecture** * 7 years - 9 years. in **Azure Databricks technologies.** * * Experience in **Medallion Architecture**is mandatory. • Good understanding of ETL , BI and DW technologies. • Hands-on experience on **Jason or XML files** as sources. • Experience in **autoloader for incremental loading and explode options for flattening data** • Experience in loading **Delta live tables** • Experience in **PySpark** or **Spark**using data frames. • Design, implement, and maintain data pipelines for data ingestion, processing, and transformation in Azure. • Work together with data scientists and analysts to understand the needs for data and create effective data workflows. • Create and maintain data storage solutions including Azure SQL Database, Azure Data Lake, and Azure Blob Storage. • Utilizing Azure Data Bricks to create and maintain **ETL**(Extract, Transform, Load) operations. • Implementing data validation and cleansing procedures will ensure the quality, integrity, and dependability of the data. • Improve the scalability, efficiency, and cost-effectiveness of data pipelines. • Monitoring and resolving data pipeline problems to guarantee consistency and availability of the data. • Good communication skills -- Oral \& Written