Tcs Hiring For Pyspark Data Engineer

Job description

Job Description PySpark Data Engineer Job Title: PySpark Data Engineer Experience: 5-10 Years Location: Chennai Job Summary We are looking for a skilled PySpark Data Engineer to design, develop, and optimize large-scale data processing pipelines. The ideal candidate should have strong expertise in PySpark, Spark ecosystem, SQL, ETL development, and cloud/big data technologies. Key Responsibilities • Develop and optimize data pipelines using PySpark and Apache Spark. • Perform data ingestion, transformation, and processing of large datasets. • Design scalable ETL/ELT solutions for data warehousing and analytics. • Work with distributed computing frameworks and big data technologies. • Integrate data from multiple sources and ensure data quality. • Collaborate with Data Scientists, Analysts, and Business teams. • Troubleshoot performance issues and optimize Spark jobs. • Follow best practices for data governance, security, and code quality. Required Skills • Strong hands-on experience in PySpark development. • Expertise in Spark SQL, DataFrames, and Spark optimization techniques. • Experience with Hadoop ecosystem (Hive, HDFS, Kafka). • Good knowledge of SQL and database concepts. • Experience in ETL pipeline development. • Knowledge of Azure Databricks, Azure Data Factory, or AWS EMR is preferred. • Experience with Git, CI/CD, and Agile methodologies.