Data Eng (Python+Pyspark+SQL+ETL)

Job description

**Job Title: Data Engineer (Python, PySpark, SQL, ETL)Role & responsibilities** **Preferred candidate profile** We are seeking a skilled and motivated **Data Engineer** with strong expertise in **Python, PySpark, SQL, and ETL development** to design, build, and maintain scalable data pipelines and data processing solutions. The ideal candidate will have experience working with large datasets, data warehouses, and cloud-based data platforms, ensuring high-quality, reliable, and efficient data delivery for analytics and business intelligence needs. Key Responsibilities - Design, develop, and maintain scalable ETL/ELT pipelines using Python, PySpark, and SQL. - Extract, transform, and load data from multiple source systems into data lakes and data warehouses. - Develop and optimize PySpark jobs for large-scale data processing. - Write complex SQL queries, stored procedures, and performance-tuned database solutions. - Implement data quality checks, validation rules, and monitoring processes. - Collaborate with business analysts, data scientists, and stakeholders to understand data requirements. - Perform root cause analysis and resolve data-related production issues. - Optimize data models, partitioning strategies, and query performance. - Maintain documentation of data flows, transformations, and technical designs. - Support batch and near real-time data processing requirements. Required Qualifications - Bachelor's degree in Computer Science, Information Technology, Engineering, or related field. - **6+ years** of experience in Data Engineering or ETL Development. - Strong programming experience in **Python**. - Hands-on experience with **PySpark** and Apache Spark framework. - Advanced knowledge of **SQL** and relational databases. - Experience designing and implementing ETL workflows. - Strong understanding of Data Warehousing concepts, including Fact and Dimension modeling. - Experience working with Parquet, ORC, JSON, and CSV data formats. - Knowledge of version control systems such as Git. - Strong analytical and problem-solving skills. Preferred Qualifications - Experience with **Azure Databricks**, Azure Data Factory (ADF), ADLS, Synapse, or AWS data services. - Knowledge of Delta Lake and Lakehouse architecture. - Experience with workflow orchestration tools such as Airflow. - Understanding of CI/CD practices and DevOps concepts. - Familiarity with streaming technologies such as Kafka or Spark Streaming. Technical Skills **Must Have:** - Python - PySpark - SQL - ETL Development - Data Warehousing - Performance Tuning **Good to Have:** - Azure Databricks - Azure Data Factory (ADF) - ADLS - Apache Airflow - Delta Lake - Git - Kafka Experience - 6-10 Years of relevant experience in Data Engineering. Responsibilities in Daily Operations - Develop and enhance data pipelines. - Monitor and troubleshoot ETL jobs. - Ensure data integrity and consistency. - Participate in code reviews and design discussions. - Work closely with cross-functional teams to deliver data solutions.