Big Data Engineer

Job description

**Software Engineer Big Data with 4+ yrs to 15 Years of experience** 4+ years of professional experience in Big data platform. Responsible for producing excellent quality of code, adhering to expected coding standards and industry best practices. **Key Responsibilities:** - Design, develop and optimize big data processing pipelines using Spark/Scala and Hive (HQL). - Implement and manage data ingestion workflows using Apache NiFi. - Develop data streaming solutions leveraging Apache Kafka. - Monitor and analyse logs and metrics using Kibana, Elasticsearch, and Logstash. - Collaborate with DevOps teams to implement Continuous Integration and Continuous Deployment (CI/CD) pipelines using Git, Jenkins, and Nexus. - Ensure data quality, consistency, and governance across big data platforms. - Participate in architecture discussions and recommend best practices for data processing and pipeline design. - Troubleshoot and resolve complex data-related issues across the big data stack. - Mentor junior engineers and contribute to team knowledge sharing. - Stay updated with emerging big data technologies and recommend adoption as appropriate - The candidate should have been in client facing roles and interacted with customers in requirement gathering workshops. - Experience working in Agile development environments and following Agile methodologies. - Should have completed at least 2 implementation projects on the above-mentioned mandatory skills and should be an independent contributor. **Skill Set:** - SPARK/SCALA - HQL - Shell Scripting / Python - Postgresql - Scheduling – Oozie and Control M - CD/CI concepts & Technologies - Big Data Ecosystem - Git & Jenkins, Nexus, Ansible, Maven, Sonar - NIFI, Kafka – Good to have - Kibana, Elastic Search Log stash – Good to have