OSS Infrastructure & Apache Kafka- Senior Developer

Job description

Company Profile: Founded in 1976, CGI is among the largest independent IT and business consulting services firms in the world. With 94,000 consultants and professionals across the globe, CGI delivers an end-to-end portfolio of capabilities, from strategic IT and business consulting to systems integration, managed IT and business process services and intellectual property solutions. CGI works with clients through a local relationship model complemented by a global delivery network that helps clients digitally transform their organizations and accelerate results. CGI Fiscal 2024 reported revenue is CA$14.68 billion and CGI shares are listed on the TSX (GIB.A) and the NYSE (GIB). Learn more at cgi.com. Job Title: OSS Infrastructure & Apache Kafka- Senior Developer Position: Senior Systems Engineer Experience: 5+ yrs Category: IT Infrastructure Main location: Hyderabad - Position ID: J0626-0615 Employment Type: Full Time Qualification: Bachelor's degree in Computer Science or related field or higher with minimum 3 years of relevant experience. Job Description: Apache Kafka Architecture & Deployment . Design and deploy multi-broker Apache Kafka clusters for high-throughput, low-latency financial messaging pipelines . Implement geo-redundant and multi-datacenter Kafka topologies using MirrorMaker 2 or Confluent Replicator . Configure KRaft mode (ZooKeeper-free) deployments and manage migration from ZooKeeper-based clusters . Define and enforce topic partitioning strategies, replication factors, and retention policies aligned with data governance requirements . Tune JVM heap settings, OS-level I/O parameters, and network buffers for optimal broker performance Resiliency & High Availability . Implement fault-tolerant cluster configurations with appropriate ISR (In-Sync Replica) thresholds and unclean leader election policies . Design and test disaster recovery (DR) runbooks, including broker failure, rack failure, and full-datacenter failover scenarios . Configure Kafka consumer group rebalancing strategies to minimize consumer lag during rolling restarts and upgrades . Establish lag monitoring, dead-letter queue (DLQ) patterns, and automated alerting using Prometheus, Grafana, or Datadog . Perform capacity planning and load testing to ensure SLA adherence under peak financial transaction volumes Security & Compliance . Configure end-to-end TLS/mTLS encryption for inter-broker and client-to-broker communication . Implement SASL authentication (Kerberos/GSSAPI, SCRAM-SHA-512, OAuth/OIDC) and fine-grained ACL authorization . Apply network segmentation, firewall rules, and broker listener isolation for financial-grade zero-trust architectures . Ensure compliance with financial regulations including PCI-DSS, SOX, GDPR, and FINRA data handling requirements . Integrate Kafka with enterprise secret management solutions (HashiCorp Vault, CyberArk) for credential rotation . Conduct security audits of Kafka configurations and remediate findings in line with CIS Benchmarks and DORA/NIST frameworks Open Source Ecosystem & Integration . Install, configure, and manage the broader OSS data platform stack, including: . Apache Zookeeper / KRaft . Kafka Connect (source & sink connectors for databases, S3, Elasticsearch) . Kafka Streams and/or Apache Flink for real-time stream processing . Schema Registry (Confluent or Apicurio) for Avro/Protobuf/JSON Schema enforcement . Automate infrastructure provisioning using Ansible, Terraform, or Helm for Kubernetes-based deployments . Integrate Kafka with CI/CD pipelines and support blue/green and canary deployment models Operations & Governance . Establish and maintain runbooks, architecture decision records (ADRs), and operational documentation . Participate in on-call rotation and provide Level 3 support for production Kafka incidents . Collaborate with application development, security, and compliance teams to define producer/consumer contracts and SLAs . Conduct code and configuration reviews; mentor junior engineers on OSS best practices Experience . 7+ years of professional software engineering experience in enterprise or financial services environments . 5+ years of hands-on Apache Kafka administration, deployment, and performance tuning in production . Demonstrated experience deploying OSS platforms at scale within regulated industries (banking, capital markets, insurance, or payments) . Proven track record of implementing Kafka security hardening (TLS, SASL, ACLs) in PCI-DSS or SOX-compliant environments Technical Skills . Apache Kafka: broker configuration, topic management, consumer group internals, log compaction, quotas . Security: mTLS, SASL/GSSAPI (Kerberos), SCRAM, OAuth 2.0, PKI certificate lifecycle management . Infrastructure as Code: Ansible, Terraform, Helm, or equivalent . Container orchestration: Kubernetes (Strimzi Kafka Operator experience a strong plus) . Monitoring & observability: Prometheus, JMX Exporter, Grafana, ELK/OpenSearch, or Splunk . Linux systems administration: RHEL/CentOS/Ubuntu, kernel tuning, systemd, network configuration . Programming/scripting: Python, Bash, or Java (for Kafka Streams / custom connectors) . Source control and DevOps: Git, Jenkins, GitLab CI, or GitHub Actions Knowledge Areas . Distributed systems fundamentals: consensus algorithms, CAP theorem, exactly-once semantics (EOS) . Financial messaging standards and protocols (FIX, SWIFT, ISO 20022) — familiarity preferred . Data governance concepts: schema evolution, data lineage, retention and archival policies . Network security in financial environments: DMZ architecture, bastion hosts, private endpoints PREFERRED QUALIFICATIONS . Confluent Certified Administrator for Apache Kafka (CCAAK) or equivalent certification . Experience with Confluent Platform (Contro