KafKa Admin

Job description

- Provision, configure, and manage Kafka clusters (Apache Kafka KRaft or ZooKeeper-backed if legacy), Confluent Platform components (Schema Registry, Kafka Connect, ksqlDB, REST Proxy, Control Center), and Confluent Cloud resources. - Ensure high availability (HA) and resilience across multi‑AZ/region deployments; manage partition replication, min.insync.replicas (ISR), and rack awareness. - Implement and maintain client quotas, broker quotas, throttling, and back‑pressure strategies. - Plan and execute upgrades/patching with zero/minimal downtime; maintain version currency against EOL and security advisories. - Own backup/restore workflows for metadata (e.g., cluster configs), Schema Registry, and Connect configs - Implement authentication (mTLS/SASL: SCRAM, OAUTHBEARER with IdP, or Kerberos if legacy) and authorization (Kafka ACLs, Confluent RBAC). - Enforce encryption in transit and at rest, secret management (Vault/AKV/SSM), secure endpoint exposure, and private connectivity (VPC peering/PrivateLink). - Govern topic naming conventions, retention policies, schema evolution rules (backward/fully compatible), and PII handling. - Maintain audit trails (broker, Schema Registry, Control Center, Confluent Cloud audit logs) and compliance artifacts (SOX, GDPR, HIPAA/PCI as applicable). - Coordinate vulnerability management (CVE watch, hardening baselines, CIS/STIG alignment). - Deploy and maintain metrics pipelines (JMX → Prometheus/Grafana), logs (ELK/OPensearch), and tracing (OpenTelemetry where applicable). - Define and monitor SLOs/SLIs (produce/consume latency, end‑to‑end lag, broker CPU/heap/file handles, network throughput, GC pauses). - Analyze broker hotspots, partition skew, large message handling, compaction and retention settings, and tune GC/JVM for sustained throughput. - Implement consumer lag monitoring and alerting with actionable runbooks to prevent data loss or SLA breaches. - Model workload growth, topic/partition scaling, storage/IOPS, network egress/ingress, and broker sizing. - For Confluent Cloud, optimize environment → cluster → topic hierarchy, throughput tiers, retention costs, and data flow egress; right‑size Dedicated vs. Standard clusters. - Design partitioning strategies to meet throughput targets while balancing consumer concurrency and avoiding small partitions anti‑patterns - Automate cluster and topic lifecycle using Terraform (providers for Kafka/Confluent), Helm, GitOps workflows. - Operate and scale Kafka Connect with distributed workers; manage connectors (JDBC, Debezium CDC, S3/ADLS/GCS, Elasticsearch, etc.), transforms (SMTs), and converter settings. - Administer Schema Registry (compatibility levels, subject naming strategies, serializers/deserializers, schema size & references). - Operate ksqlDB and/or Flink for streaming SQL; enforce resource limits and multi‑tenant governance. - Engineer low‑latency and secure connectivity across data centers/VPCs/VNETs; manage DNS, TLS SANs, LB configurations, advertised.listeners, and inter‑cluster networking. **Required Skills & Experience** - 5–10 years in platform/SRE/DevOps; 5+ years dedicated to Kafka/Confluent admin in production. - Strong hands‑on with Apache Kafka internals: brokers, controllers (KRaft), partitions/replication, log segments, compaction/retention. - Confluent Platform: Schema Registry, Kafka Connect, ksqlDB, Control Center, Confluent REST APIs; or Confluent Cloud admin (environments, clusters, RBAC, audit logs, networking). - Proficiency with Kafka CLI tools and AdminClient usage; deep knowledge of ACLs, quotas, transactions, idempotent producers, and exactly‑once semantics scenarios. - Prometheus/Grafana, JMX exporters, alerting systems (Alertmanager etc). - JVM tuning, Linux performance (NUMA, file descriptors, page cache, disk/RAID, XFS), and network tuning (TCP buffers, MTU, jumbo frames where applicable). - Linux administration (systemd, journald, kernel params), virtualization/containers (Docker, Kubernetes). - Cloud networking (VPC/VNET peering, transit gateways, PrivateLink/Private Service Connect), multi‑region architectures. **Preferrable -** - Confluent Certified Administrator for Apache Kafka (CCAAK), CKA/CKAD, HashiCorp Terraform Associate, major cloud certs (Azure/AWS/GCP). - Experience with Confluent for Kubernetes (CFK) for operator‑based deployments.