Sr. Manager - Data Engineer

Job description

TransUnion's Job Applicant Privacy Notice Team Overview An M02 is responsible for managing a development team, ensuring delivery of software solutions, and aligning execution with business and product goals. The role balances: * People management * Technical oversight * Delivery accountability Typically owns a single team of engineers

This is a hybrid position and involves regular performance of job responsibilities virtually as well as in-person at an assigned TU office location for a minimum of two days a week.Role Overview And Core Responsibilities Team Leadership & Delivery • Lead a team of software and data engineers (typically 5–10 members). • Ensure timely and high-quality delivery of platform capabilities, data processing solutions, and feature enhancements. • Drive execution of large-scale distributed processing initiatives and data platform modernization efforts. • Foster a culture of engineering excellence, innovation, and accountability. Technical Oversight • Provide technical leadership for large-scale distributed data processing systems built on Apache Spark. • Guide architecture, design, and implementation decisions for scalable, fault-tolerant, and high-performance data processing workloads. • Establish best practices for Spark application development, optimization, observability, and operational excellence. • Ensure adherence to coding standards, performance requirements, scalability guidelines, and security controls. • Lead troubleshooting and resolution of complex production issues involving Spark jobs, data pipelines, cluster performance, and resource management. • Drive optimization of Spark workloads through partitioning strategies, query tuning, caching, memory management, and efficient resource utilization. Data Platform & Spark Engineering • Design and oversee development of batch and real-time data processing pipelines using Apache Spark. • Collaborate on architecture involving Spark,  Iceberg, Hive, Hadoop,  AWS EMR, AWS Glue, GCP Dataproc, BigQuery, and other modern cloud-native data platform technologies. • Ensure reliability, scalability, and operational efficiency of enterprise-scale data workflows. • Drive adoption of data engineering best practices, CI/CD automation, testing frameworks, and monitoring for Spark workloads. • Champion performance benchmarking, capacity planning, and cost optimization initiatives across data processing platforms. Stakeholder Collaboration • Work closely with product managers, architects, data scientists, platform teams, and business stakeholders. • Translate business and data processing requirements into executable engineering plans. • Communicate delivery status, technical risks, architectural decisions, and mitigation plans to leadership. • Partner with cross-functional teams to deliver data-driven products and platform capabilities. People Management • Conduct performance reviews, coaching, and regular feedback sessions. • Mentor engineers in distributed systems, Spark development, performance optimization, and software engineering best practices. • Build technical depth within the team through knowledge sharing and career development. • Support hiring, onboarding, and growth of engineering talent with expertise in big data and distributed computing. Process & Quality Management • Ensure Agile, SDLC and engineering governance practices are followed. • Track delivery, quality, reliability, and operational metrics. • Improve engineering productivity through automation, standardization, and platform improvements. • Drive continuous improvement initiatives focused on system reliability, performance, and customer satisfaction. Required Knowledge And Experiences Required Knowledge and Experience Technical Expertise • Strong hands-on experience with Apache Spark (Spark Core, Spark SQL, Structured Streaming, DataFrames, Dataset APIs). • Proven experience building and operating large-scale distributed data processing systems. • Strong understanding of Spark performance tuning, partitioning strategies, joins, shuffles, caching, memory management, and resource optimization. • Experience with one or more programming languages such as Scala, Java, or Python. • Experience with modern data ecosystems including Hadoop, Hive, Iceberg, AWS EMR, AWS Glue, GCP Dataproc, BigQuery, or equivalent technologies. • Experience designing and maintaining batch and streaming data pipelines. • Strong understanding of distributed systems, cloud-native architectures, and scalability principles. • Hands-on experience deploying and managing Spark workloads on AWS and/or GCP. Leadership Experience • Proven experience leading engineering teams and delivering complex technical initiatives. • Ability to balance hands-on technical leadership with people management responsibilities. • Experience driving cross-functional collaboration and influencing technical direction. • Demonstrated success managing technical roadmaps, execution planning, and delivery commitments. Scope & Positioning • Mid-level management role positioned between Senior Engineer/Technical Lead and