Lead Big Data Engineer

Job description

About the Role: Grade Level (for internal use): 11The Team: Our enterprise data management team is at the forefront of designing and developing cutting-edge data solutions that drive strategic business outcomes across the organization. We value quick learners who thrive in a dynamic technology environment and can seamlessly transition between collaborative teamwork and independent problem-solving. The team fosters a culture of continuous learning and innovation, where members tackle emerging technologies while maintaining strong analytical rigor and effective cross-functional communication. Responsibilities and Impact: • Design and develop scalable data processing pipelines using distributed computing frameworks to handle large-scale enterprise datasets • Build and maintain ETL processes that extract, transform, and load data into strategic information products supporting organizational goals • Develop data-intensive applications and services using modern programming languages and cloud-based analytics platforms • Implement stream processing solutions using real-time data processing technologies to support business-critical operations • Collaborate with cross-functional teams to translate complex business requirements into robust technical solutions • Provide production support and troubleshooting for data systems, ensuring high availability and performance What We're Looking For: Basic Required Qualifications: • 8+ years of hands-on experience in technology application development and production support • 6+ years of experience developing data pipelines that extract, transform, and load data using programming languages such as Python, Scala, or Java • Minimum 3+ years of experience developing and supporting ETL processes using cloud-based analytics platforms (such as Databricks, Snowflake, or Azure Synapse) • Experience building data-intensive applications using modern programming technologies (including but not limited to C#, Java, Python, Scala, and SQL) • Proficiency with cloud computing environments (such as AWS, Azure, or Google Cloud Platform) Additional Preferred Qualifications: • Experience with stream processing technologies (such as Apache, Apache Pulsar, or Amazon Kinesis) for real-time data processing • Hands-on experience with distributed processing frameworks (such as Apache Spark, Dask, or Hadoop MapReduce) for large-scale data analytics • Knowledge of data integration tools (such as Apache NiFi, Talend, or Informatica) and database replication technologies • Experience with CI/CD pipelines, containerization technologies (such as Docker, Podman, or containerd), and orchestration platforms (such as Kubernetes, Docker Swarm, or OpenShift) About S&P Global RatingsAt S&P Global Ratings, our analyst-driven credit ratings, research, and sustainable finance opinions provide critical insights that are essential to translating complexity into clarity so market participants can uncover opportunities and make decisions with conviction. By bringing transparency to the market through high-quality independent opinions on creditworthiness, we enable growth across a wide variety of organizations, including businesses, governments, and institutions. S&P Global Ratings is a division of S&P Global (NYSE: SPGI).  S&P Global is the world’s foremost provider of credit ratings, benchmarks, analytics and workflow solutions in the global capital, commodity and automotive markets. With every one of our offerings, we help many of the world’s leading organizations navigate the economic landscape so they can plan for tomorrow, today.For more information, visit www.spglobal.com/ratings What’s In It For You? Our Mission: Advancing Essential Intelligence. Our People: We're more than 35,000 strong worldwide—so we're able to understand nuances while having a broad perspective. Our team is driven by curiosity and a shared belief that Essential Intelligence can help build a more prosperous future for us all.From finding new ways to measure sustainability to analyzing energy transition across the supply chain to building workflow solutions that make it easy to tap into insight and apply it. We are changing the way people see things and empowering them to make an impact on the world we live in. We’re committed to a more equitable future and to helping our customers find new, sustainable ways of doing business. Join us and help create the critical insights that truly make a difference. Our Values: Integrity, Discovery, Partnership Throughout our history, the world's leading organizations have relied on us for the Essential Intelligence they need to make confident decisions about the road ahead. We start with a foundation of integrity in all we do, bring a spirit of discovery to our work, and collaborate in close partnership with each other and our customers to achieve shared goals.Benefits: We take care of