Databricks with Python and Cloud

Job description

## **Your Role** - 7+ years of experience in data engineering, data platforms & analytics, 10+ years of consulting experience - Minimum 6-8+ projects delivered with hands-on experience in development on Databricks - Strong hands-on experience in Python programming. Expertise in: - - Pandas (data manipulation, transformation, analysis) - NumPy (array operations, numerical computing) - Design, develop, and optimize scalable data pipelines on Databricks using Apache Spark (PySpark/Scala), ensuring high performance and reliability for batch and real-time processing. - Implement data engineering best practices including Delta Lake, data modeling, partitioning, and performance tuning; manage data workflows using Databricks Workflows or orchestration tools. - Collaborate with data scientists and analysts to build and deploy machine learning models and analytics solutions, leveraging Databricks notebooks, MLflow, and Unity Catalog for governance. ## **Your Profile** - Ensure data quality, security, and compliance by applying monitoring, logging, access controls, and CI/CD pipelines (Azure DevOps/Git), supporting end-to-end data lifecycle management - Working knowledge of two or more common Cloud ecosystems (AWS, Azure, GCP) with deep expertise in at least one - Deep experience with distributed computing with Spark with knowledge of Spark runtime internals - Familiarity with CI/CD for production deployments - Working knowledge of MLOps - Current knowledge across the breadth of Databricks product and platform features - Familiarity with optimizations for performance and scalability - Databricks certification is an added advantage ## **What will you love working at Capgemini** - You will have the opportunity to learn on one of the industry's largest digital learning platforms, with access to 250,000+ courses and numerous certifications. - We’re committed to ensure that people of all backgrounds feel encouraged and have a sense of belonging at Capgemini. You are valued for who you are, and you can bring your original self to work. - At Capgemini, you can work on cutting-edge projects in tech and engineering with industry leaders or create solutions to overcome societal and environmental challenges. - Capgemini office campuses in India are green and run on 100% renewable electricity. We have installed Solar plants across India locations and ‘Battery Energy Storage Solution’ (BESS) in the Noida and Mumbai campuses. You will have chance to make a difference everyday.