Senior AWS Data Engineer

Job description

**About Position:** We are looking for a highly skilled AWS Data Engineer with strong expertise in AWS, Databricks, and PySpark to build and optimize scalable data platforms and AI-driven solutions. The ideal candidate will have extensive experience in designing modern data architectures, performance tuning, implementing MLOps practices, and leveraging Databricks Mosaic AI capabilities. - **Role: Senior AWS Data Engineer** - **Location: All Persistent Location** - **Experience: 10+ Years** - **Job Type: Full-Time Employment** **What You'll Do:** - As a Senior AWS Data Engineer, you will be responsible for designing, developing, and optimizing enterprise-scale data solutions while enabling advanced AI/ML initiatives. - Design, develop, and maintain scalable data pipelines using AWS and Databricks. - Build and optimize ETL/ELT workflows using PySpark and Spark SQL. - Develop data solutions based on Lakehouse architecture and Delta Lake. - Perform Spark and Databricks performance tuning to improve processing efficiency and reduce costs. - Implement and maintain MLOps workflows for model deployment, monitoring, and lifecycle management. - Leverage Databricks Mosaic AI capabilities for Generative AI and ML use cases. - Collaborate with Data Scientists, ML Engineers, Architects, and business stakeholders. - Ensure data quality, governance, security, and compliance standards are followed. - Automate data engineering processes using CI/CD pipelines and Infrastructure as Code. - Troubleshoot and resolve complex production issues across data platforms. **Expertise You'll Bring:** - The ideal candidate should possess strong technical expertise in cloud-based data engineering, big data technologies, and AI/ML ecosystems. - 10+ years of overall IT experience with strong Data Engineering background. - Hands-on experience with AWS services including S3, Glue, EMR, Lambda, Redshift, Athena, IAM, and Step Functions. - Extensive experience working with Databricks, Delta Lake, Unity Catalog, and Databricks Workflows. - Strong development experience using PySpark and Spark SQL. - Expertise in Spark Performance Tuning, Query Optimization, Partitioning, and Cluster Management. - Experience implementing MLOps frameworks using MLflow and CI/CD methodologies. - Hands-on experience with Databricks Mosaic AI, LLMs, RAG, and Generative AI solutions. - Strong proficiency in Python and SQL. - Experience with version control tools such as Git. - Knowledge of Terraform, Docker, Kubernetes, and Airflow will be an added advantage. - Strong analytical, problem-solving, and communication skills. - Ability to work independently and collaborate with cross-functional teams. - Bachelor's or master's degree in computer science, Information Technology, Data Science, Engineering, or a related field. - Relevant AWS and Databricks certifications are highly preferred. - AWS Certified Data Engineer – Associate - AWS Certified Solutions Architect - Databricks Certified Data Engineer Professional - Databricks Generative AI Engineer Certification - Databricks Machine Learning Professional Certification **Benefits:** - Competitive salary and benefits package - Culture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications - Opportunity to work with cutting-edge technologies - Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards - Annual health check-ups - Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents **Values-Driven, People-Centric & Inclusive Work Environment:** Persistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds. - We support hybrid work and flexible hours to fit diverse lifestyles. - Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities. - If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment **Let's unleash your full potential at Persistent -** **persistent.com/careers** **"Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind."**