GCP Data Engineer - Pyspark (PAN India)

Job description

**Key Responsibilities** - Design and implement modern data engineering solutions on Google Cloud Platform. - Develop and maintain scalable batch and real-time data processing pipelines. - Build cloud-based data lakes and modern data warehouse solutions. - Create and maintain optimal data pipeline architecture using GCP services. - Work with: - BigQuery - Dataflow - Dataproc - Cloud Storage - Data Fusion - Airflow - Drive data platform modernization and transformation initiatives. - Perform assessment and analysis of legacy data platforms and define modernization roadmaps. - Implement process improvements for: - Data ingestion - ETL processing - Data delivery - Platform scalability - Develop extraction, transformation, and loading solutions using Python and PySpark. - Support CI/CD implementation for data engineering deployments. - Define and monitor KPIs for platform modernization initiatives. - Collaborate with business, product, analytics, and engineering teams. **Must-Have Skills** Google Cloud Platform - Strong hands-on experience with: - BigQuery - Cloud Storage (GCS) - Dataflow - Dataproc - Data Fusion - Airflow - Experience working on cloud-native data platforms. Data Engineering - Strong understanding of: - Data Warehousing - Data Lakes - Data Platform Modernization - ETL/ELT Frameworks Programming - Strong expertise in: - Python - PySpark - SQL Big Data - Experience building: - Batch processing solutions - Real-time data pipelines - Knowledge of Spark-based distributed data processing. DevOps & Automation - Exposure to: - CI/CD Pipelines - Release Automation - Monitoring & Logging Frameworks Operating Systems - Strong knowledge of: - Linux **Good-to-Have Skills** - Experience with: - Hadoop - Hive - Kafka - Apache Beam - Sqoop - Exposure to: - On-Premise to Cloud migrations - Enterprise Data Modernization programs