JobScans.in

Job description

Job Description We are looking for an experienced **Data Engineer** with strong expertise in **PySpark, Python, SQL, and Azure Databricks** to design, develop, and optimize scalable data pipelines and big data solutions. Key Responsibilities - Design, build, and maintain scalable data pipelines and data processing solutions. - Work with large datasets using **PySpark, Spark, and SQL**. - Develop and optimize data models for analytics and reporting. - Implement batch and real-time data processing workflows. - Collaborate with cross-functional teams to deliver high-quality data solutions. - Perform performance tuning and optimization of Spark jobs and SQL queries. - Ensure data quality, testing, and deployment best practices. Required Skills - 5+ years of experience in **Data Engineering** or Big Data technologies. - Strong hands-on experience in **Python, PySpark, Spark, and SQL**. - Experience with **Azure Databricks, Azure Data Lake, and Azure Data Factory**. - Knowledge of data modeling concepts and large-scale data processing. - Experience building and optimizing ETL/ELT pipelines. - Understanding of cloud-based data platforms and architectures. Nice to Have - Experience with CI/CD pipelines and DevOps practices. - Knowledge of GitLab, PowerShell, Shell/Bash scripting. - Exposure to Power BI or other BI reporting tools. - Experience working in Agile/Scrum environments. - Financial domain experience is a plus. **Keywords:** Data Engineer, Big Data, PySpark, Spark, Python, SQL, Azure Databricks, Azure Data Factory, Azure Data Lake, ETL, Data Modeling, Cloud Data Engineering.