Azure Databricks Platform Engineer

Talent Join

India, Telangana, Hyderabad

On-site · Full-time

About the role

We are looking for a Senior Azure Databricks Platform Engineer with strong hands-on experience in Databricks development and platform operations. You will independently manage and support the Databricks platform in conjunction with Azure cloud services. Workspace creation, maintenance, and deployment using DevOps CI/CD are integral to the role.

Responsibilities

  • Design, develop, and optimize data pipelines using Azure Databricks, Python, PySpark, Spark SQL, and Delta Lake.
  • Develop and maintain Databricks notebooks, workflows, and scheduled jobs.
  • Configure and manage Databricks clusters, cluster policies, autoscaling, runtime versions, and libraries.
  • Support Databricks workspace configuration, infrastructure management, and platform operations.
  • Implement Unity Catalog access controls, data governance, and data lineage.
  • Support Azure security, Key Vault integration, and secrets management.
  • Implement CI/CD pipelines and automate Databricks deployments.
  • Monitor and troubleshoot Databricks jobs, clusters, connectivity, and production issues.
  • Optimize Spark performance, compute resources, and platform costs.
  • Collaborate with Data Engineering, DevOps, and Infrastructure teams.

Must-have requirements

  • 5+ years of experience in platform engineering on Azure data platforms, including at least 3 years of Azure Databricks platform management.
  • End-to-end understanding of Databricks, from developing Python/PySpark solutions to configuring clusters, managing jobs, and supporting production environments. Experience with workspace management, cluster management, and Unity Catalog implementation.
  • Strong Python fundamentals, including data structures, functions, error handling, and problem-solving.
  • Hands-on experience with Azure Databricks, PySpark, Spark SQL, SQL, ADLS Gen2, and Delta Lake.
  • Practical experience developing Databricks notebooks, workflows, and job scheduling.
  • Hands-on experience with Databricks cluster management, autoscaling, runtime configuration, and cluster policies.
  • Experience with Unity Catalog, access management, and data lineage implementation and monitoring.
  • Good understanding of Databricks infrastructure, workspace configuration, and Azure cloud services.
  • Experience with CI/CD pipelines, deployment automation, and DevOps practices.
  • Knowledge of Azure networking, security, Key Vault, and secrets management.
  • Experience troubleshooting production Databricks jobs, clusters, and platform issues.
  • Experience with Terraform for provisioning, configuring, and managing Azure Databricks infrastructure, including workspaces, clusters, Unity Catalog, and related Azure resources.
  • Good understanding of Spark architecture and performance optimization.
  • Strong communication and problem-solving skills.

Nice to have

  • Experience with Bicep, ARM Templates, or other Infrastructure as Code tools.
  • Experience with Databricks Asset Bundles (DABs).
  • Experience with Azure Monitor, Log Analytics, and cost optimization.
  • Experience provisioning Databricks workspaces from scratch.
  • Experience in banking, insurance, or other regulated industries.
  • Azure or Databricks certifications.