Hiring Technical Lead Data Engineering | Kafka, AWS, PySpark, Scala

Job description

**We are Hiring Technical Lead Data Engineering | Kafka, AWS, Spark (Scala/PySpark), Kubernetes/EKS** **Company:** Coforge Ltd. **Job Location: Hyderabad, Pune & Greater Noida Only.** **Experience:** 8 to 12 Years **Employment Type:** Full-Time **Key Skills:-** Kafka Must Have, Spark, Scala, PySpark, AWS, S3, Glue, EMR, Lambda, Kinesis, Athena, Redshift, EKS, Kubernetes, Docker, Helm, Python, SQL, Terraform, CI/CD, ETL, ELT, Real-Time Data Pipelines, Data Engineering, Technical Leadership. **Share your CV:** **Gaurav.2.Kumar@coforge.com** **If you have any questions regarding this opportunity, please feel free to contact us on WhatsApp at:** **9667427662 ( Gaurav Kumar HR, Coforge Ltd )** **Role Overview:-** We are seeking an experienced **Technical Lead Data Engineering** to lead a team of Data Engineers in designing, developing, and supporting large-scale, cloud-native data platforms. The ideal candidate will possess strong expertise in **Kafka, AWS Data Engineering services, Spark (Scala/PySpark), and Kubernetes/EKS**, along with proven experience in technical leadership, architecture, and stakeholder management. The role requires hands-on technical leadership while driving engineering best practices, mentoring team members, and ensuring successful delivery of scalable, secure, and high-performance data solutions. **Key Responsibilities:-** **Technical Leadership** - Lead and mentor a team of Data Engineers across design, development, testing, and production support. - Drive architecture and technical design decisions for enterprise-scale data platforms. - Establish engineering standards, coding practices, CI/CD processes, and operational excellence. - Conduct code reviews and provide technical guidance to team members. - Collaborate with architects, product owners, and business stakeholders to translate requirements into scalable solutions. **Data Engineering & Platform Development** - Design and develop real-time and batch data pipelines using Kafka, Spark, Scala, and PySpark. - Build scalable data processing frameworks on AWS. - Develop and optimise ETL/ELT pipelines for high-volume structured and unstructured datasets. - Ensure data quality, governance, observability, and operational monitoring. - Troubleshoot performance bottlenecks and optimise data processing workloads. **Cloud & Kubernetes** - Deploy and manage data workloads on Amazon EKS (Elastic Kubernetes Service). - Implement containerised applications using Docker and Kubernetes. - Develop infrastructure automation and deployment pipelines. - Work closely with DevOps teams to implement CI/CD and infrastructure-as-code practices. **Stakeholder & Delivery Management:-** - Participate in sprint planning, estimation, and technical roadmaps. - Manage technical risks, dependencies, and delivery commitments. - Provide status updates to leadership and stakeholders. - Support production issues and critical incident resolution. **Required Skills** **Data Engineering:-** - Strong experience with **Apache Kafka** including producers, consumers, Kafka Streams, schema management, and event-driven architectures. - Expertise in **Apache Spark** using **Scala and PySpark**. - Experience designing and supporting real-time and batch data processing solutions. - Strong understanding of distributed systems and big data concepts. **AWS Data Engineering -** Hands-on experience with: - AWS S3 - AWS Glue - AWS EMR - AWS Lambda - AWS Kinesis - AWS Athena - AWS Redshift - AWS IAM - CloudWatch - Step Functions **Kubernetes & Containers:** - Strong expertise in **Amazon EKS** - Kubernetes administration and deployment strategies - Docker containerisation - Helm Charts - Autoscaling and resource optimisation **Programming:** - Scala - Python - SQL - Shell Scripting **DevOps:** - Git/GitHub/GitLab - Jenkins/Azure DevOps/GitHub Actions - Terraform (preferred) - CI/CD pipelines **Leadership Expectations:-** - Lead distributed teams and provide technical mentorship. - Drive solution design discussions and architecture reviews. - Ensure adherence to security, compliance, and operational standards. - Foster innovation and continuous improvement within the team. - Act as the primary technical escalation point for critical issues. **Preferred Qualifications:-** - Experience working in Banking/Financial Services environments. - Exposure to Data Lakehouse architectures. - Experience with Delta Lake, Iceberg, or Hudi. - Knowledge of data governance and data quality frameworks. - AWS Certifications (Solutions Architect, Data Engineer, Developer Associate). - Kubernetes Certification (CKA/CKAD) is a plus. **Join This WhatsApp Group for Regular and Quick Job Updates.** **Click On the Link:-** **https://chat.whatsapp.com/LW8Dnu6IZ54HzkZVMakCE7**