Gcp Data Engineer

Job description

**Senior Data Engineer - GCP, Big Query, PySpark & Kafka** **Location** Hyderabad (Hybrid) **Experience** 5 to 13 Years **Role Overview** We are looking for a highly skilled **Data Engineer** with strong expertise in building scalable, cloud-native data platforms on **Google Cloud Platform (GCP)**. The ideal candidate should have extensive hands-on experience with **Python, PySpark, BigQuery, Kafka, DBT, Terraform, Kubernetes, SQL, CI/CD, and ETL pipeline development**. The candidate will be responsible for designing, developing, optimizing, and supporting large-scale data processing solutions that drive business insights and analytics. **Key Responsibilities** - Design, develop, and maintain scalable data pipelines and data processing frameworks on GCP. - Build and optimize batch and real-time ETL/ELT pipelines using Python, PySpark, Kafka, and BigQuery. - Develop high-performance data solutions for large-scale structured and unstructured datasets. - Create and maintain data models using DBT and ensure adherence to data governance standards. - Design streaming solutions using Kafka and event-driven architectures. - Implement infrastructure as code (IaC) using Terraform for automated cloud provisioning. - Deploy and manage containerized applications using Kubernetes. - Build automated CI/CD pipelines for data platform deployments and releases. - Optimize BigQuery performance, partitioning, clustering, and query tuning. - Collaborate with data scientists, analysts, architects, and business stakeholders to build data products. - Troubleshoot production issues and drive continuous improvements in platform reliability and performance. - Ensure data quality, scalability, security, and operational excellence across the data ecosystem. **Must Have Skills** **Data Engineering** - Python - PySpark / Apache Spark - Advanced SQL - ETL / ELT Pipeline Development - Data Warehousing Concepts - Data Modeling **Google Cloud Platform (GCP)** - BigQuery - Dataflow - Pub/Sub - Cloud Composer - Cloud Storage (GCS) - Cloud Functions (Good to Have) **Streaming & Messaging** - Apache Kafka - Event-Driven Architecture - Real-Time Data Processing **DevOps & Cloud Engineering** - Terraform - Kubernetes - Docker - CI/CD Pipelines - Git/GitHub/GitLab **Data Transformation** - DBT (Data Build Tool) **Required Qualifications** - Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or related field. - 5-13 years of experience in Data Engineering and Cloud-based Data Platforms. - Strong hands-on expertise in GCP ecosystem and BigQuery. - Experience building enterprise-grade data platforms and large-scale distributed data processing systems. - Strong understanding of cloud-native architectures and modern data engineering practices. - Excellent problem-solving and analytical skills. - Strong communication and stakeholder management capabilities. **Preferred Skills** - Experience with Data Lake and Lakehouse architectures. - Knowledge of Airflow/Cloud Composer. - Exposure to BI/Analytics platforms such as Tableau, Looker, or Power BI. - Experience in Financial Services, Banking, Insurance, or Enterprise Data Platforms. - Exposure to Data Governance and Data Quality tools. **What We Are Looking For** - Strong hands-on coding experience in Python and PySpark. - Deep expertise in BigQuery performance optimization. - Proven experience with Kafka-based streaming solutions. - Strong Terraform and Kubernetes implementation experience. - Experience building robust CI/CD pipelines for data platforms. - Ability to design end-to-end cloud-native data solutions on GCP.