Lead Data Engineer

Job description

**Job Title:** Manager - Data Management **Department** : ISPL Data Office, CIB IT Transversal **About Business line/Function:** The Server Infrastructure team serves as a pivotal component of BNP Paribas' regional Hub in India, tasked with the supervision, monitoring, and development of the server infrastructure that underpins the bank's regional operations. In close collaboration with BNP Paribas Head Offices, where most major banking applications are hosted, the team plays a vital role in shaping the processes that inform decision-making around IT investments, foster client and supplier relationships, and drive project management initiatives. By leveraging their technical expertise and strategic insights, the Server Infrastructure team contributes significantly to the bank's ability to optimize its IT infrastructure and ensure seamless service delivery ultimately supporting the growth and success of our Organization BNP Paribas. **Position Purpose:** The DPR (Data Platform Referential) team is responsible for providing technical and functional support to ensure the reliable operation of applications for a specific Business Line or application chain. This includes managing IT assets for infrastructure, such as IAAS, PAAS, CAAS, and IT cloud services. The lead maintains the service in operational condition, ensuring smooth process execution, including batch processing and data transfer, as well as proper application functioning with relevant business data and correct workflow execution. In a rapidly changing environment, the DPR team member / lead manages releases within their scope, responding to incidents and implementing preventive measures through data analysis and automation to enhance service quality and user experience. As experts, they collaborate with IT Operations on production and infrastructure requirements, serving as a guarantor of production throughout the project cycle. The lead plays a key role in incorporating production requirements into project planning, contributing to the definition of technical solutions, ensuring the stability of technical environments, and monitoring technical solutions. He/She participates in project team meetings, engage in collective decision-making, and assesses complexity and risks throughout the project. **Responsibilities** Direct Responsibilities - Design and develop real-time and batch data pipelines using tools like Apache NiFi, Apache Kafka, and Apache Airflow. - Implement and maintain ETL/ELT processes, ensuring data quality and integrity across various data sources. - Integrate and manage data flows across diverse systems using NiFi and Kafka topics. - Monitor and optimize the performance of data pipelines and troubleshoot issues proactively. - Work with the ELK stack (Elasticsearch, Logstash, Kibana) for logging, monitoring, and real-time analytics. - Write efficient and optimized SQL queries for data extraction, transformation, and reporting. - Collaborate with data analysts and business stakeholders to develop meaningful Power BI dashboards and reports. - Maintain documentation for data workflows, processes, and architecture. **Required Skills** - Strong experience with Apache NiFi for data flow management. - Expertise in Kafka (streaming data, Kafka Connect, schema management). - Proficiency in Apache Airflow for orchestrating workflows. - Solid understanding of SQL (writing complex queries, tuning, indexing). - Experience working with the ELK stack – Elasticsearch for search and analytics, Logstash for ingestion, Kibana for visualization. - Experience with Power BI – data modeling, DAX, dashboard development. - Knowledge of data warehousing concepts, data modeling, and data lake architecture. Contributing Responsibilities The Digital Platform domain within the CIB ITO PRODUCTION entity is in charge of the following scopes: - IAAS (Infrastructure As A Service): Server / OS / Storage / Backup / Virtualization - CAAS (Container As A Service): Containers / Middleware / Sharepoint - PAAS (Platform As A Service): Databases / Data Analtics - Infrastructure Services: Micro segmentation / Load Balancing / API gateway / Automation - Risk & Improvement: Obsolescence / Security / Continuous improvement / ITIL Process - Work as a member of a team, adhere to corporate standards and change management processes, and work to improve IT standards and policies. - Proficient in English. - Good interpersonal and communication skills. Competency **Business Skills** - Architecture (Mastered) - Knowledge of Data (Mastered) - Computer tests (Expert) - IT infrastructure (Expert) **Transversal Skills** - Analytical capacity (Expert) - Ability to lead a meeting, a seminar, a committee, a training session, etc. (Mastered) - Ability to understand, explain and lead change (Mastered) - Ability to define relevant performance indicators (Mastered) - Ability to work with Agile methods (Mastered) **Behavioral Skills** - Ability to share/transmit knowledge (Expert) - Be results oriented (Expert) - Creativity & Innovation / Ability to solve problems (Expert) - Ability to collaborate / teamwork (Expert) - Develop and maintain system documentation, including configuration guides, and standard operating procedures. - Direct and be responsible for the implementation effort. - Provide technical guidance and mentorship to team members. - Assess demand for their service or technology area and develop plans to meet future capacity needs and make recommendations to the manager. - Aware of all critical changes to infrastructure and applications that could impact service delivery to their business customers. - Able to work autonomously and as part of a team using strong analytical skills. - Be service oriented, customer focused, positive, committed and have an enthusiastic “can do” attitude. - Demonstrate a systematic and logical approach to problem-solving. - Able to follow the bank’s standards, processes