Azure Databricks with Pyspark

Job description

**Job Title: - Azure Databricks with Pyspark** **Location: - AIA Kolkata** **Experience: - 6 to 9 Years** An Azure Databricks Developer job description typically requires expertise in PySpark, Azure Data Factory (ADF), and Delta Lake architecture **Key Responsibilities** · Pipeline Development: Build and optimize scalable data pipelines and ETL processes using Azure Databricks and PySpark. · Data Integration: Connect and manage data from Azure Data Lake Storage (ADLS Gen2) and various file formats (Parquet, Delta, JSON, CSV). · Performance Tuning: Tune Spark SQL queries, optimize cluster configurations, and manage data partitioning strategies. · CI/CD \& Deployment: Implement version control via Git and deploy automated workflows using Azure DevOps or GitHub Actions. **Required Qualifications** · Experience: 4 to 8 years of total IT experience, with at least 2--3 years of hands-on production experience in Azure Databricks. · Tech Stack: Advanced proficiency in Python, PySpark, and complex SQL. · Cloud Knowledge: Strong familiarity with core Azure services including ADF, ADLS Gen2, and Azure SQL. Cognizant is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to sex, gender identity, sexual orientation, race, color, religion, national origin, disability, protected Veteran status, age, or any other characteristic protected by law.