Gcp Data Engineer

Job description

**Role & responsibilities** **Job Overview** We are seeking a skilled **GCP Data Engineer** to design, build, and optimize our data pipelines and architecture on Google Cloud Platform. In this role, you will work with cross-functional teams to build scalable, real-time, and batch data processing systems using modern GCP tools. You will be responsible for transforming raw data into actionable insights for downstream analytics and machine learning applications. **Key Responsibilities** - **Pipeline Design & Development:** Build, deploy, and maintain scalable batch and real-time ETL/ELT pipelines using **GCP Dataflow** (Apache Beam) and **Apache Airflow** (Cloud Composer). - **Data Warehousing:** Design, optimize, and manage large-scale data warehouses and data marts in **Google BigQuery**, ensuring high performance and cost efficiency through effective partitioning, clustering, and SQL optimization. - **Stream & Event Processing:** Ingest and process high-throughput real-time streaming data using **Google Cloud Pub/Sub**. - **Big Data Processing:** Configure, manage, and execute distributed data processing workloads using **Google Cloud Dataproc** (Apache Spark/Hadoop). - **Data Preparation & Quality:** Clean, transform, and wrangle complex structured and unstructured datasets using **Google Cloud Dataprep** and custom **Python** scripts. - **Code & Query Development:** Write efficient, maintainable **Python** code and complex **SQL** queries for data ingestion, transformation, and validation. - **Optimization & Monitoring:** Monitor pipeline performance, troubleshoot failure points, and optimize execution time and cloud infrastructure costs. - **Collaboration:** Partner with Data Analysts, Data Scientists, and Software Engineers to understand data requirements and deliver business-ready datasets. **Required Qualifications & Skills** - **Hands-on GCP Experience:** Solid experience building end-to-end data pipelines using **Google Cloud Platform** native tools. - **Core Languages:** Advanced proficiency in **Python** and standard ANSI/Google Standard **SQL**. - **Data Warehousing:** Expertise in **BigQuery** architecture, query optimization, view creation, and cost management. - **Orchestration & ETL:** Strong experience orchestrating workflows using **Apache Airflow** and building transformations with **Dataflow** or **Dataprep**. - **Big Data & Streaming:** Proven experience handling streaming architectures with **Pub/Sub** and big data cluster processing via **Dataproc**. - **Data Modeling:** Solid understanding of relational database design, data modeling concepts (Star Schema, Snowflake Schema), and data lake architectures. **Preferred candidate profile** - Google Cloud Certified Professional Data Engineer certification. - Knowledge of DevOps principles, CI/CD pipelines, and Infrastructure as Code (Terraform). - Experience with Git, Docker, and Linux environments.