Specialist, Data Engineering

Job description

**Job Description** **Specialist, Data Engineer** **The Opportunity** - Based in Hyderabad, join a global healthcare biopharma company and be part of a 130- year legacy of success backed by ethical integrity, forward momentum, and an inspiring mission to achieve new milestones in global healthcare. - Be part of an organisation driven by digital technology and data-backed approaches that support a diversified portfolio of prescription medicines, vaccines, and animal health products. - Drive innovation and execution excellence. Be a part of a team with passion for using data, analytics, and insights to drive decision-making, and which creates custom software, allowing us to tackle some of the world's greatest health threats. Our Technology Centers focus on creating a space where teams can come together to deliver business solutions that save and improve lives. An integral part of our company's IT operating model, Tech Centers are globally distributed locations where each IT division has employees to enable our digital transformation journey and drive business outcomes. These locations, in addition to the other sites, are essential to supporting our business and strategy. A focused group of leaders in each Tech Center helps to ensure we can manage and improve each location, from investing in growth, success, and well-being of our people, to making sure colleagues from each IT division feel a sense of belonging to managing critical emergencies. And together, we must leverage the strength of our team to collaborate globally to optimize connections and share best practices across the Tech Centers. **ROLE Overview** We are looking for a data & platform engineer to join the team responsible for the Data configuration-driven data pipeline solution that powers data ingestion, transformation, and delivery across AWS Glue, Databricks, and Apache Airflow. In addition to the core data platform, this role owns AI agents that enable users to automate configuration creation and pipeline troubleshooting and provide guided technical support. This is an individual contributor role spanning the full stack a Python/PySpark pipeline engine, Scala/Java Spark extensions, GitHub Actions CI/CD workflows, multi-cloud infrastructure. You will own features and releases end-to-end, support internal teams consuming the framework, and keep the platform secure, scalable, and reliable. **What You Will Do** - Develop and maintain the core engine — build and extend Python/PySpark loaders, transformers, and writers across 23 source connectors and 19 sink connectors. - Drive CI/CD automation — design, maintain, and improve 28+ GitHub Actions reusable workflows covering dataset build/deploy, framework releases, Docker image promotion, and AWS key rotation. - Manage multi-cloud infrastructure — provision and maintain AWS resources (ECS, IAM, ECR, S3, Secrets Manager) and Azure using Terraform. - Own and evolve the product Configuration and Support Bot — maintain the Microsoft Teams bot adapter, integrate with the Company approved LLM, and extend features such as file attachment handling, channel thread context, and Microsoft Graph integration. - Automate credential lifecycle — operate and improve the automated AWS IAM key rotation service that keeps GitHub Actions secrets, Airflow connections, and AWS Secrets Manager in sync. - Ensure data quality — implement and extend the rule engine for schema validation, null checks, regex patterns, and quarantine/alert actions. - Support internal consumers — help dataset teams onboard, troubleshoot pipelines, and adopt new framework features; maintain API stability across releases. - Contribute to release management — own versioning strategy and artifact promotion through JFrog Artifactory for core, orchestration, and tooling packages. - Provide L3 technical support for end users. **What You Should Have** Data Engineering **4+ years** of professional experience in data engineering, platform engineering, or cloud infrastructure - Strong experience with Python and PySpark for batch and streaming ETL workloads - Proficiency in Spark SQL, DataFrame API, and custom Spark extensions - Experience with data lake patterns Delta Lake, Parquet, partitioning strategies - Understanding of data quality, schema validation, and Change Data Capture (CDC) - Familiarity with data orchestration platforms (e.g. Apache Airflow,) - Experience integrating diverse data sources relational databases (JDBC), REST APIs with OAuth2, cloud object storage, file transfer protocols (SFTP/SMB) or streaming systems Cloud Platforms & Infrastructure as Code - Hands-on experience with AWS (compute, storage, data warehousing, messaging, identity, container services, monitoring) and Azure (compute, app hosting, identity, networking) - Strong Infrastructure as Code skills — Terraform or equivalent tooling for provisioning, managing, and tearing down cloud resources across environments - Experience managing multi-environment deployments (dev / test / production) with proper isolation and promotion workflows - Container management building, tagging, promoting, and hosting container images in cloud registries - Familiarity with cloud cost management and resource right-sizing Cloud Networking & Security - Understanding of cloud networking VPC/VNet design, subnet architecture, private endpoints - Identity and access management across cloud providers IAM policies, service principals, app registrations, role-based access control CI/CD & DevOps - Proficiency in GitHub Actions or comparable CI/CD platforms reusable workflows, matrix build strategies, environment-scoped secrets - Docker and containerization best practices for build and deployment pipelines - Artifact and release management versioning strategies, artifact promotion, dependency management across multiple packages - Experience with artifact repositories (e.g. JFrog Artifactory, Nexus, or cloud-native equiva