PySpark+ Databricks(7-14yrs)

Job description

**Roles and responsibility** - Design, develop, and maintain scalable data pipelines using PySpark and Databricks. - Build and optimize ETL/ELT processes for large-scale data processing. - Develop data transformation and data integration workflows in Databricks. - Work with structured and unstructured data from various sources. - Optimize Spark jobs for performance, scalability, and cost efficiency. - Implement Delta Lake architecture and manage data quality. - Collaborate with data analysts, data scientists, and business stakeholders. - Troubleshoot and resolve data pipeline and performance issues. - Ensure adherence to data governance and security standards. - Participate in code reviews, testing, and deployment activities. Required Skills - Strong experience in **PySpark** and **Apache Spark**. - Hands-on experience with **Databricks** workspace and notebooks. - Proficiency in **Python** and **SQL**. - Experience with **Delta Lake**, Spark SQL, and Spark Optimization techniques. - Knowledge of data warehousing concepts and ETL frameworks. - Experience with cloud platforms such as **Azure, AWS, or GCP**. - Understanding of data modeling and big data technologies. - Experience with version control tools like Git.