Azure Databricks Platform Engineer
Talent Join
India, Telangana, Hyderabad
On-site · Full-time
About the role
We are looking for a Senior Azure Databricks Platform Engineer with strong hands-on experience in Databricks development and platform operations. You will independently manage and support the Databricks platform in conjunction with Azure cloud services. Workspace creation, maintenance, and deployment using DevOps CI/CD are integral to the role.
Responsibilities
- Design, develop, and optimize data pipelines using Azure Databricks, Python, PySpark, Spark SQL, and Delta Lake.
- Develop and maintain Databricks notebooks, workflows, and scheduled jobs.
- Configure and manage Databricks clusters, cluster policies, autoscaling, runtime versions, and libraries.
- Support Databricks workspace configuration, infrastructure management, and platform operations.
- Implement Unity Catalog access controls, data governance, and data lineage.
- Support Azure security, Key Vault integration, and secrets management.
- Implement CI/CD pipelines and automate Databricks deployments.
- Monitor and troubleshoot Databricks jobs, clusters, connectivity, and production issues.
- Optimize Spark performance, compute resources, and platform costs.
- Collaborate with Data Engineering, DevOps, and Infrastructure teams.
Must-have requirements
- 5+ years of experience in platform engineering on Azure data platforms, including at least 3 years of Azure Databricks platform management.
- End-to-end understanding of Databricks, from developing Python/PySpark solutions to configuring clusters, managing jobs, and supporting production environments. Experience with workspace management, cluster management, and Unity Catalog implementation.
- Strong Python fundamentals, including data structures, functions, error handling, and problem-solving.
- Hands-on experience with Azure Databricks, PySpark, Spark SQL, SQL, ADLS Gen2, and Delta Lake.
- Practical experience developing Databricks notebooks, workflows, and job scheduling.
- Hands-on experience with Databricks cluster management, autoscaling, runtime configuration, and cluster policies.
- Experience with Unity Catalog, access management, and data lineage implementation and monitoring.
- Good understanding of Databricks infrastructure, workspace configuration, and Azure cloud services.
- Experience with CI/CD pipelines, deployment automation, and DevOps practices.
- Knowledge of Azure networking, security, Key Vault, and secrets management.
- Experience troubleshooting production Databricks jobs, clusters, and platform issues.
- Experience with Terraform for provisioning, configuring, and managing Azure Databricks infrastructure, including workspaces, clusters, Unity Catalog, and related Azure resources.
- Good understanding of Spark architecture and performance optimization.
- Strong communication and problem-solving skills.
Nice to have
- Experience with Bicep, ARM Templates, or other Infrastructure as Code tools.
- Experience with Databricks Asset Bundles (DABs).
- Experience with Azure Monitor, Log Analytics, and cost optimization.
- Experience provisioning Databricks workspaces from scratch.
- Experience in banking, insurance, or other regulated industries.
- Azure or Databricks certifications.