Data Engineer (Pyspark +Databricks+AWS )

Job description

Skill: Data Engineer (Databricks + Pyspark) Loc: Pune Mode: 5 days WFO Shift: UK shift – 2:30 to 11 PM Exp: 8-13 Years **JD** We are looking for a skilled **Databricks Data Engineer** to design, develop, and optimize scalable data pipelines and data platforms. The ideal candidate will have strong expertise in **Databricks, SQL, Python, and PySpark** , along with experience in implementing **CI/CD practices** for data engineering workflows. **Primary Skills (Mandatory)** 1. **Databricks** - Strong hands-on experience with Databricks (Data Lakehouse architecture) - Expertise in Delta Lake, Unity Catalog, and Databricks Workflows - Experience in performance tuning and cost optimization 1. **SQL** - Advanced SQL skills for data transformation, optimization, and querying - Experience in writing complex joins, window functions, and performance tuning 1. **Python** - Proficient in Python for data engineering tasks and scripting - Experience with modular code development and reusable components 1. **PySpark** - Strong experience in building scalable data pipelines using PySpark - Knowledge of Spark optimization techniques (partitioning, caching, etc.) 1. **CI/CD** - Experience implementing CI/CD pipelines for data workflows - Familiarity with tools such as Azure DevOps, GitHub Actions, Jenkins, or similar - Knowledge of version control (Git) and automated deployment of data solutions **Secondary Skills** - **AWS** - Experience working with AWS services such as S3, EC2, IAM, and EMR - **Apache Airflow** - Experience in workflow orchestration and scheduling using Airflow - **AWS Glue** - Hands-on experience in ETL processing and Glue jobs **Good to Have** - **Terraform** - Knowledge of Infrastructure as Code (IaC) for provisioning cloud resources - Experience in automation and environment setup using Terraform **Key Responsibilities** - Design and develop scalable data pipelines using Databricks and PySpark - Build and optimize ETL/ELT processes for large-scale data processing - Collaborate with data architects and stakeholders to define data models and solutions - Implement CI/CD pipelines for automated deployment and testing - Monitor, troubleshoot, and optimize data workflows for performance and cost - Ensure data quality, governance, and security standards are maintained - Work closely with cross-functional teams across data, analytics, and business domains **Required Experience** - 8+ years of experience in data engineering or related roles - Hands-on experience in Databricks-based data platform projects - Strong understanding of distributed data processing concepts