Software Development Engineer

Job description

**Role & responsibilities** **Key Responsibilities** - Design, develop, and maintain end-to-end data pipelines and ETL/ELT processes using Databricks and other data engineering tools to support analytics and BI use cases. - Build and deploy cloud-native data and analytics solutions on AWS, leveraging services such as EKS, S3, and other relevant components for scalable and resilient architectures. - Develop backend services and APIs using Python and Flask to enable data access, automation workflows, and integration with downstream applications and dashboards. - Implement containerization and orchestration using Docker and Kubernetes (EKS preferred) to ensure reliable, portable, and scalable deployment of data and analytics workloads. - Set up and maintain CI/CD pipelines using Jenkins/Jenkins Core to automate build, test, and deployment processes for data engineering and analytics solutions. - Collaborate with data scientists, analysts, and BI developers to enable AI, machine learning, and advanced analytics use cases through well-structured, high-quality data. - Optimize data models, queries, and pipelines for performance, cost efficiency, and reliability across large-scale datasets and complex analytics workloads. - Implement data quality checks, monitoring, logging, and alerting to ensure data reliability, integrity, and timely availability for business stakeholders. - Contribute to data automation initiatives by identifying opportunities to streamline manual processes and implementing robust automation solutions. - Ensure adherence to best practices in cloud security, data governance, and compliance for all data engineering and analytics solutions. - Support troubleshooting, root cause analysis, and performance tuning for data pipelines, cloud infrastructure, and analytics platforms. - Mentor junior team members and contribute to knowledge sharing, coding standards, and best practices within the data engineering and analytics team.