Azure Databricks Platform Admin

Job description

## **Key Responsibilities:** - Azure Databricks Platform Administration - Administer and manage Azure Databricks workspaces clusters and jobs - Configure and maintain Databricks environments Dev QA Prod - Monitor cluster performance job execution and system health - Manage notebooks libraries and dependencies - Perform upgrades patches and platform maintenance - Cluster Management Optimization - Design and manage cluster configurations auto scaling instance types pools - Optimize job performance resource utilization and execution times - Troubleshoot cluster failures memory issues and job bottlenecks - Implement best practices for Spark performance tuning - Security Governance - Implement and enforce data security and access controls RBAC ACLs IAM - Manage workspace security secrets Key Vault integration and credential passthrough - Ensure compliance with enterprise security and governance policies - Implement data governance auditing and lineage tracking - Azure Cloud Integration - Integrate Databricks with Azure services such as - Azure Data Lake Storage ADLS Gen2 - Azure Key Vault - Azure Data Factory ADF - Azure Synapse Analytics - Manage connectivity and networking VNet Private Endpoints NSGs ## **Technical Requirements:** - Core Skills - 10 15 years of IT experience with strong Azure Databricks administration - Hands on expertise in Apache Spark and Databricks platform management - Strong experience with Azure cloud services ADLS ADF Key Vault Networking - Proven experience in production support and platform operations - Technical Skills - Deep knowledge of cluster configuration job scheduling and performance tuning - Strong understanding of Spark architecture and optimization techniques - Experience with Terraform ARM Bicep IaC tools - Knowledge of security models IAM RBAC and data governance - Familiarity with CI CD tools and DevOps practices - Preferred Skills - Experience with Delta Lake and Lakehouse architecture - Exposure to Python PySpark Scala - Familiarity with monitoring tools Azure Monitor Splunk - Knowledge of multi cloud environments AWS GCP - Experience in data engineering or ETL pipelines ## **Preferred Skills:** Technology->Cloud Platform->Azure Analytics Services->Azure Databricks