Python PySpark Developer

Job description

**Key Skills** **PySpark, Python** **Experience -5-10 years** **Location:- Chennai and Bengaluru** **Must have Skills** - Implementing data ingestion pipelines from different types of data sources i.e Databases, S3, Files etc.. - Experience in building ETL/ Data Warehouse transformation process. - Experience working with structured and unstructured data. - Developing Big Data and non-Big Data cloud-based enterprise solutions in PySpark and SparkSQL and related frameworks/libraries, - Developing scalable and re-usable, self-service frameworks for data ingestion and processing, - Integrating end to end data pipelines to take data from data source to target data repositories ensuring the quality and consistency of data, - Processing performance analysis and optimization, - Bringing best practices in following areas: Design & Analysis, Automation (Pipelining, IaC), Testing, Monitoring, Documentation. **Good to have (Knowledge)** 1. Experience in cloud-based solutions, 2. Knowledge of data management principles.