Senior/Lead Data Engineer

Job description

**Responsibilities-** 1. Data Engineering & Pipeline Development Monitor and optimize compute usage across workloads. • Evaluate storage patterns, caching techniques, and data partitioning. • Design, build, and maintain scalable ETL/ELT pipelines using Databricks (PySpark, Spark SQL, Delta Lake). • Architect data ingestion frameworks to process structured, semi-structured, and unstructured data. • Develop re-usable frameworks for data quality, data validation, and exception handling. 2. Data bricks & Cloud Platform Expertise • Optimize Data bricks clusters, jobs, and queries for performance and cost efficiency. • Manage cluster configurations, auto scaling, job scheduling, and deployment workflows. • Implement best practices for notebook modularization, CI/CD, and version control. 3. Data Architecture & Modelling • Design end-to-end data models including lakehouse architecture, medallion layers (bronzesilvergold). • Collaborate with Data Science team to prepare training datasets and feature engineering pipelines. • Implement governance, cataloguing, and lineage using cloud-native tools. 4. Performance & Cost Optimization • Recommend architecture and platform-level improvements to reduce cost. 5. Collaboration & Documentation • Work closely with cross-functional teams including Data Science, Analytics, Business, and IT Infra. • Maintain detailed documentation for pipelines, data flows, and technical designs. • Support production deployments, monitoring, and troubleshooting. **Educational Qualifications** Graduation/Post Graduation **Competencies Required: –** • Strong hands-on knowledge of Databricks, Spark, Delta Lake, and PySpark. • Experience in architecting and scaling cloud-based pipelines in Azure • Strong SQL expertise and understanding of distributed data processing. • Experience with source versioning (Git), CI/CD, and DevOps workflows. • Proficiency with REST APIs, Databricks SQL Warehouses, and Unity Catalog. • Good understanding of data warehousing concepts, lakehouse patterns, and metadata management. • Exposure to ML model deployment workflows • Intermediate proficiency in Power BI (dataset preparation, modeling, dashboards). **Experience -** 5+ years of experience as a Data Engineer or in a similar role.