Gcp Data Engineer

Job description

Data Engineering & Pipeline Development • Design, develop, and maintain scalable batch and real-time data pipelines on Google Cloud Platform (GCP). • Build reliable and efficient data ingestion, transformation, and processing frameworks using BigQuery, Dataflow, Dataproc, BigTable, Pub/Sub, Cloud Storage, and Data Fusion. • Create and operationalize data pipelines by integrating multiple enterprise data sources while ensuring data quality, consistency, and availability. • Develop prototype analytics pipelines to generate business insights and support product innovation. AI/ML & Predictive Analytics • Build, train, and optimize machine learning models for VIN-level predictive risk scoring, loss ratio forecasting, and high-utilization contract identification. • Perform advanced Exploratory Data Analysis (EDA) on historical claims, contracts, and operational datasets from sources such as OWS, PTS, UDB, and DMS. • Engineer business-driven features, including complex metrics such as Claim vs. Unclaimed Learnings and High Time In Service (HTIS). • Support AI/ML solution design, experimentation, and prototype development using Python and GCP services. MLOps & Model Lifecycle Management • Manage the complete machine learning lifecycle, including: • Data preprocessing and feature engineering • Model training and validation • Deployment and productionization • Performance monitoring and retraining • Implement MLOps best practices to ensure model scalability, reliability, and maintainability. • Monitor model performance and proactively address model drift and data quality issues. Cloud Platform & Architecture • Design, build, secure, monitor, and optimize data processing systems on Google Cloud Platform. • Develop expertise in Google technologies, architectures, and data structures to support future product development and business initiatives. • Implement cloud-native solutions following scalability, security, and cost-optimization best practices. Data Quality, Testing & Governance • Perform unit testing, integration testing, and validation of data pipelines and analytical solutions. • Identify and resolve defects, data inconsistencies, and performance bottlenecks. • Establish data quality checks, monitoring frameworks, and governance standards. • Ensure compliance with organizational policies, ethical AI practices, and data privacy regulations. DevOps & Automation • Utilize Git, Jenkins, Terraform, Tekton, and CI/CD pipelines to automate deployments and infrastructure management. • Support Infrastructure as Code (IaC) practices for cloud resource provisioning and configuration management. • Implement automation to improve deployment efficiency, reliability, and repeatability. Collaboration & Agile Delivery • Collaborate with business stakeholders, product owners, data scientists, and engineering teams to understand requirements and deliver data-driven solutions. • Participate actively in Agile ceremonies, including: • Sprint Planning • Backlog Grooming & Prioritization • Daily Standups • Sprint Reviews • Retrospectives • Provide technical recommendations and contribute to product roadmap discussions. Technical Skills • Strong proficiency in Python, SQL, and Google Cloud Platform (GCP). • Experience with Big Data technologies and distributed data processing frameworks. • Knowledge of machine learning, predictive analytics, feature engineering, and model deployment. • Familiarity with DevOps, CI/CD, monitoring, and cloud security best practices.