Data bricks Lead Data Engineer – R01565550

Job description

Data bricks Lead Data Engineer - R01565550 Lead Data Engineer Primary Skills • ETL Fundamentals, SQL, SQL (Basic + Advanced), Python, Data Warehousing, Modern Data Platform Fundamentals, Data Modelling Fundamentals, PLSQL, T-SQL, Stored Procedures Specialization • Databricks Engineering: Data Engineer Job requirements Job Description – Data Engineer (Databricks) Company: Brillio Technologies Role: Data Engineer – Databricks Experience: 5–10 Years Location: India (Hybrid/3 days) Employment Type: Full-time About Brillio Brillio is a leading digital transformation partner delivering innovative solutions across Data, Cloud, AI, and Digital Engineering. We help global enterprises unlock value through cutting-edge technology and data-driven insights. Role Overview We are looking for a hands-on Data Engineer with strong Databricks expertise to design, build, and scale modern data platforms. The ideal candidate will have experience in PySpark, Delta Lake, and cloud-native data engineering and will work closely with analytics, product, and business teams to deliver high-performance data pipelines. Key Responsibilities • Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and PySpark • Build and optimize data lakehouse architectures (Bronze–Silver–Gold layers) • Develop data ingestion frameworks from multiple sources (batch & real-time) • Implement Delta Lake features including schema evolution, ACID transactions, and optimization • Create and manage Databricks notebooks, workflows, and jobs for production pipelines • Ensure data quality, governance, and lineage across pipelines • Optimize Spark jobs for performance (partitioning, caching, tuning) • Collaborate with cross-functional teams (Data Scientists, Analysts, Architects) • Support production deployments, monitoring, and troubleshooting of data pipelines • Contribute to migration of legacy data platforms to modern cloud-based architecture Technical Skills (Must-Have) • Strong experience in: • - Databricks (Workflows, Notebooks, Jobs) • PySpark / Spark SQL • SQL & Data Warehousing concepts • Hands-on experience with: • - Delta Lake / Lakehouse architecture • Data pipeline development & orchestration • Good understanding of: • - Data modeling, ETL/ELT processes, SCD concepts Cloud & Tools • Experience in at least one: • - AWS (S3, Glue, EMR) / Azure (ADF, ADLS, Synapse) • Workflow tools: • - Airflow / Databricks Workflows • Version control: • - Git / CI-CD pipelines Good to Have • Experience with: • - Streaming frameworks (Kafka / Spark Streaming) • DBT / Snowflake / Redshift / Fabric • Exposure to: • - Unity Catalog, Autoloader, Delta Live Tables Experience Required • 5–10 years of experience in Data Engineering • Minimum 2–4 years of hands-on Databricks experience • Proven experience building enterprise-scale data platforms and pipelines Soft Skills • Strong problem-solving and analytical skills • Ability to work in a fast-paced, agile environment • Good communication and stakeholder management Why Join Brillio • Work on cutting-edge data & AI programs • Opportunity to build large-scale data platforms • Collaborative and innovation-driven culture