SENIOR DATA ENGINEER - Databricks

Job description

Senior Data Engineer (68 years) Role Overview Hands-on IC role focused on building reliable, performant data pipelines on Azure Databricks, implementing Delta Lake patterns, and contributing to production-grade data products for analytics. Core Responsibilities Build/maintain batch basic streaming pipelines using PySpark/Spark SQL and Delta Lake following Medallion principles. Ingest via Autoloader, ADF, REST, and file sources (e.g., SFTP/SharePoint). Apply schema enforcement/evolution; add basic data quality checks and validations. Use Unity Catalog for catalog/schema/table permissions; handle basic secrets via Key Vault/secret scopes. Author clear code docs; participate in PRs; contribute utilities; triage job issues and first-line performance fixes. Operate simple Databricks Workflows (or ADF) and Git-based CI/CD basics. Must-have Skills (Screening Focus) Strong hands-on PySpark, Spark SQL, Delta Lake fundamentals; solid SQL. Understanding of partitions, shuffles, joins, caching; Medallion basics. Unity Catalog basics (catalog/schema/table permissions). Autoloader basics; ADF familiarity or Workflows basics. Git workflows; basic tests and logging; runbook hygiene. Nice-to-have (Prefer, not Mandatory) Exposure to DLT and Workflows orchestration patterns. Photon awareness; cost-aware design exposure. Healthcare/PII familiarity; Power BI integrations. 90-day Outcomes (Examples) Ship 2 SilverGold pipelines with Add basic data expectations (null, duplicate, type checks) with alerting on a critical table Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.