Performance Engineer - Data Engineering Specialist

Job description

Performance Engineer Project Role : Performance Engineer Project Role Description : Diagnose issues that an in-house performance testing team has been unable to. There are five aspects to Performance Engineering: software development lifecycle and architecture, performance testing and validation, capacity planning, application performance management and problem detection and resolution. Must have skills : Data Engineering Good to have skills : Python (Programming Language), GitHub Minimum 3 year(s) of experience is required Educational Qualification : 15 years full time education Summary: As a Performance Engineer, a typical day involves investigating and resolving complex performance issues that have not been identified by the internal performance testing team. The role encompasses a broad spectrum of activities including analyzing software development processes and system architecture, validating performance through rigorous testing, planning for capacity needs, managing application performance, and swiftly detecting and addressing problems to ensure optimal system functionality. This position requires a proactive approach to identifying bottlenecks and collaborating with various teams to enhance overall system efficiency and reliability. KEY RESPONSIBILITIES Set up and configure Databricks workspaces, including cluster management, access controls, and integration with cloud identity and access management services Design and implement Medallion architecture (Bronze, Silver, and Gold layers) on cloud object storage using Delta Lake format, ensuring data quality and traceability at each layer Build and maintain ETL and ELT data pipelines using PySpark and Spark SQL within Databricks, covering ingestion, transformation, deduplication, normalisation, and standardisation Ingest data from multiple source systems including relational databases (Oracle, MySQL, SQL Server), flat files, and SaaS platforms such as Salesforce, into the Lakehouse Execute data migration from Oracle and other legacy systems, storing transformed outputs in the cloud storage Gold Layer using Databricks batch processing Orchestrate and automate data workflows using pipeline orchestration tools such as Apache Airflow, AWS Glue, or Azure Data Factory, ensuring timely, dependency-aware, and reliable data delivery Create and manage Linked Services, Datasets, and connection configurations for source and sink systems Monitor pipeline execution end-to-end, troubleshoot failures, implement corrective actions, and perform root cause analysis for recurring issues Optimise Spark jobs for performance, scalability, and cost efficiency --- including partitioning strategy, caching, and resource configuration Ensure all data is stored in Delta format with appropriate schema enforcement, ACID compliance, and incremental load patterns Manage sensitive credentials and connection strings securely using a secrets management service (e.g., AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault) Maintain data quality standards by implementing validation and reconciliation logic within pipeline workflows Collaborate with data analysts, data scientists, and product teams across global delivery environments to understand data needs and deliver trusted datasets Maintain thorough documentation of pipeline designs, data flows, architecture decisions, and operational runbooks REQUIRED SKILLS \& EXPERIENCE 3--4 years of professional experience in data engineering, with a significant and demonstrable portion of that time working on Databricks in a production environment Strong hands-on proficiency in PySpark and Python, with the ability to write clean, efficient, and maintainable transformation code Proven experience designing and implementing Medallion architecture (Bronze / Silver / Gold) on cloud object storage, with Delta Lake as the storage layer Solid understanding of Delta Lake capabilities: ACID transactions, schema evolution, time travel, and incremental ingestion patterns Hands-on experience migrating and extracting data from Oracle and other relational databases (MySQL, SQL Server) into a cloud-based Lakehouse Experience with batch data ingestion from SaaS platforms such as Salesforce Working knowledge of at least one pipeline orchestration tool --- Apache Airflow, AWS Glue, Azure Data Factory, or similar --- for scheduling and dependency management Familiarity with cloud-hosted relational databases and MySQL for structured data storage and maintenance Sound understanding of ETL and ELT design patterns and the trade-offs between them Proficiency in Git and GitHub for version control, branching, and collaborative code review Experience working within Agile or Waterfall delivery frameworks with structured sprint or milestone-based planning Strong analytical and troubleshooting skills ability to diagnose and resolve pipeline failures methodically Effective communication and collaboration skills with the ability to work across cross-functional and geographically distributed teams GOOD TO HAVE Experience with Databricks Unity Catalog for data governance, lineage tracking, and fine-grained access control Exposure to cloud secrets management integration (e.g., AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault) within Databricks secret scopes Background in legacy big data tooling --- Hadoop, Hive, or Sqoop --- particularly in the context of migration or modernisation projects Familiarity with advanced Spark optimisation techniques including broadcast joins, adaptive query execution, and partition tuning Basic understanding of data modelling concepts and dimensional or star schema design Exposure to DevOps practices for data pipelines, including CI/CD, automated testing frameworks, or infrastructure-as-code Databricks Certified Associate Developer for Apache Spark, or a relevant cloud data certification (AWS Certified Data Analytics, Microsoft Certified: Azure Data Engineer Associate, or equivalent) EDUCATION Bachelor s degree (B.Tech / B.E