Apache NIFI

Job description

**Job Requirements\\*** - Experience in data engineering/pipeline development - Strong hands-on experience with Apache NiFi (flow design, custom processors, registry) - Solid understanding of Apache Kafka (topics, partitions, consumer groups, throughput optimization) - Experience with Cloudera Data Platform (CDP) ecosystem - Working knowledge of Kubernetes and container orchestration - Experience with OpenStack or similar cloud infrastructure - Familiarity with monitoring tools (Grafana, Prometheus) - Understanding of networking concepts (DNS, load balancing, firewalls) **Key Responsibilities\\*** - Design, develop, and maintain Apache NiFi data pipelines for real-time and batch data ingestion across multiple sources and sinks - Integrate NiFi pipelines with Kafka (Logstash Kafka, Kafka Kafka) and Cloudera Data Platform (CDP) components - Architect and deploy NiFi clusters on OpenStack-based Container-as-a-Service (CaaS) environments - Configure NiFi clustering and High Availability to ensure fault tolerance and zero downtime - Perform performance tuning and load optimization of NiFi processors and Kafka topics (partitions, throughput, consumer groups) - Design and implement monitoring and alerting solutions using NiFi reporting tasks, Grafana, and Prometheus - Create and maintain deployment, networking, scaling, and configuration documentation - Collaborate with platform teams on Kubernetes/container orchestration for pipeline deployments - Implement security best practices including TLS encryption, RBAC, and Kerberos authentication - Develop and maintain CI/CD pipelines for automated NiFi flow deployment and version control - Troubleshoot production issues, perform root cause analysis, and implement preventive measures - Participate in architecture reviews and provide recommendations for pipeline scalability and resilience