Big Data Developer

Job description

Job Description: • Design, develop, and maintain data pipelines and ETL workflows using Databricks, Spark, and ADF. • Implement data lakehouse architecture and optimize data storage and retrieval for performance and cost efficiency. • Write efficient, reusable, and scalable Python code for data processing and automation tasks. • Orchestrate complex workflows using Azure Data Factory and integrate with other Azure services. • Develop and maintain Power BI dashboards and reports, ensuring accurate and timely insights for stakeholders. • Ensure data quality, security, and governance across all data solutions. • Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality data products. • Monitor and troubleshoot data pipelines, ensuring high availability and reliability. Stay updated on emerging technologies and best practices in cloud data engineering. • Experience with data lakehouse architecture, Delta Lake, and data modeling. Knowledge of CI/CD pipelines, DevOps practices, and version control (Git).