MODULE LEAD - Data Management/Mining/Collection

Job description

GCP Data Engineer (6?8 Years Experience)Role Overview We are looking for a highly skilled Data Engineer with 6?8 years of experience to design, build, and optimize scalable data pipelines and AI-driven data solutions. The ideal candidate will have hands-on expertise in **Kubernetes (GKE)**, **GCP ecosystem**, **RAG-based architectures**, and **agentic/AI pipelines**, along with experience in data processing, extraction, and integration workflows. Key Responsibilities - Design and develop **scalable data pipelines** for data extraction, transformation, and ingestion. - Build and manage **AI/ML data pipelines**, including **RAG (Retrieval-Augmented Generation)** and agent-based workflows. - Develop and orchestrate **data extraction and injection pipelines** across structured and unstructured data sources. - Deploy and manage application jobs on **Kubernetes (GKE)** for high availability and performance. - Integrate with GCP services such as **BigQuery, Cloud Storage, Pub/Sub, Dataflow, and Apigee**. - Work with **SDP (Sensitive Data Protection/DLP)** and **Model Armor** for secure data handling and compliance. - Collaborate with cross-functional teams to ensure **data governance, security, and privacy compliance**. - Build and maintain **CI/CD pipelines using GitHub** and related DevOps tools. - Optimize performance, scalability, and cost-efficiency of data and AI systems. - Troubleshoot production issues and ensure system reliability. Required Skills & ExperienceCore Technical Skills - Strong experience in **Kubernetes** and **Google Kubernetes Engine (GKE)** - Hands-on experience with **Google Cloud Platform (GCP)** services - Expertise in **data pipeline development** (batch & real-time) - Experience with **data extraction, transformation, and ingestion pipelines** - Solid understanding of **RAG architectures and vector-based retrieval systems** - Experience in building **agentic pipelines / AI-driven workflows** Platform & Tools - **GitHub** (version control, CI/CD workflows) - **Apigee** (API management and integration) - **SDP/DLP** and **Model Armor** (data security and compliance handling) - Experience with **event-driven architectures** (Pub/Sub, Eventarc preferred) Programming & Data - Strong programming skills in **Python / Pyspark** - Experience working with **SQL and NoSQL databases** - Familiarity with **BigQuery, Databricks, or Spark-based processing** Good to Have - Experience with **Vertex AI / AI platform integration** - Knowledge of **data masking, tokenization, and privacy frameworks** - Exposure to **healthcare or sensitive data ecosystems** (PII/PHI handling) - Understanding of **microservices architecture and API-driven systems**