GCP Data Engineer- Chennai, Bangalore, Hyderbad, Delhi

Job description

Job Requirements\\* • Proficiency in programming languages: Python, Java • Expertise in data processing frameworks: Apache Beam (Data Flow) • Active experience on GCP tools and technologies like Big Query, Dataflow, Cloud Composer, Cloud Spanner, GCS, DBT etc., • Data Engineering skillset using Python, SQL • Experience in ETL (Extract, Transform, Load) processes • Knowledge of DevOps tools like Jenkins, GitHub, Terraform is desirable. Should have good knowledge on Kafka (Batch/ streaming) • Understanding of Data models and experience in performing ETL design and build, database replication using Message based CDC • Familiarity with cloud storage solutions • Strong problem-solving abilities in data engineering challenges • Understanding of data security and scalability • Proficiency in relevant tools like Apache Airflow Core Skills: • GCP Services: BigQuery, Dataflow, Cloud Storage, Pub/Sub, Data Catalog, Data Fusion, AI Platform. • Programming: SQL, Python, Java/Scala (for Dataflow/Spark). • Data Concepts: Data Warehousing, ETL, Data Modeling, Streaming. • Tools: Apache Beam, Docker, Kubernetes, Git, Terraform. Key Responsibilities\\* • Design & Build Data Pipelines: Develop scalable data ingestion, transformation (ETL/ELT), and orchestration workflows using GCP services. • GCP Service Utilization: Implement solutions with BigQuery, Dataflow, Dataproc, Cloud Storage, Pub/Sub, and Cloud Composer. • Data Integration: Connect and manage data from diverse sources (APIs, databases, files). • Optimization: Improve performance, cost-efficiency, and reliability of data processes. • Collaboration: Work with data scientists, analysts, and architects to meet data needs. • Quality & Security: Ensure data integrity, security, and compliance. • Automation: Implement CI/CD and Infrastructure as Code (IaC) using tools like Terraform. • Monitoring & Support: Maintain, monitor, and troubleshoot data workloads and systems.