Data Engineer

Job description

**About Position:** We are seeking a highly skilled Data Engineer with 6 to 10 years of experience in building scalable, cloud-based data solutions using Azure, Databricks, and PySpark. The ideal candidate will be responsible for designing, developing, and optimizing enterprise-grade data pipelines, enabling data-driven decision-making across the organization. You will work closely with business stakeholders, data scientists, and analytics teams to deliver reliable, secure, and high-performance data solutions. * **Role: Data Engineer** * **Location: All Persistent Location** * **Experience: 6 to 10 Years** * **Job Type: Full-Time Employment** **What You'll Do:** * As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data solutions on the Azure cloud platform using Databricks and PySpark. You will work closely with business stakeholders, data scientists, and analytics teams to build robust data pipelines and support enterprise data initiatives. * Design, develop, and optimize scalable ETL/ELT pipelines using PySpark and Azure Databricks. * Build and maintain data ingestion frameworks for structured and unstructured data. * Develop data solutions leveraging Azure services such as Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), Azure Synapse, and Azure SQL. * Implement batch and real-time data processing solutions. * Optimize Spark workloads for performance, scalability, and cost efficiency. * Collaborate with business and technical teams to understand data requirements and deliver high-quality solutions. * Ensure data quality, governance, security, and compliance standards are met. * Monitor, troubleshoot, and resolve issues related to data pipelines and processing jobs. * Support CI/CD implementation and deployment of data engineering solutions. * Contribute to data architecture discussions and best practices. **Expertise You'll Bring:** * We are seeking a highly skilled Data Engineer with strong expertise in Databricks, PySpark, and Azure cloud technologies, capable of delivering scalable and reliable data solutions. * 4 years of experience in Data Engineering and Big Data technologies. * Strong hands-on experience with Azure Databricks and PySpark. * Proficiency in Python and SQL. * Experience with Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), Azure Synapse Analytics, and Azure SQL Database. * Strong understanding of Spark Architecture, Data Lake, and Lakehouse concepts. * Experience in developing and optimizing ETL/ELT pipelines. * Knowledge of Delta Lake and data modeling concepts. * Experience with Git, Azure DevOps, and CI/CD pipelines. * Familiarity with data governance, security, and monitoring practices. * Strong analytical, problem-solving, and debugging skills. * Excellent communication and stakeholder management skills. * Experience working in Agile/Scrum environments. **Benefits:** * Competitive salary and benefits package * Culture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications * Opportunity to work with cutting-edge technologies * Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards * Annual health check-ups * Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents **Values-Driven, People-Centric \& Inclusive Work Environment:** Persistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds. * We support hybrid work and flexible hours to fit diverse lifestyles. * Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities. * If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment **Let's unleash your full potential at Persistent -** **persistent.com/careers** **"Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind."**