Senior Data Engineer (Databricks & AI Solutions)

Cephas Consultancy Services Private Limited

India, Karnataka, Bengaluru | India, Tamil Nadu, Chennai | India, Telangana, Hyderabad

On-site · Full-time

We are seeking a highly skilled and motivated Senior Data Engineer with 6+ years of experience to design, develop, and support scalable data platforms and streaming pipelines. This role leverages modern Databricks capabilities, including Lakeflow and Unity Catalog, alongside AI and generative AI technologies. The ideal candidate will bring strong hands-on expertise in building robust data engineering solutions, implementing governance standards, and delivering high-quality data products for business-critical use cases.

Key Responsibilities

  • Design, develop, and maintain end-to-end streaming and batch data pipelines using Databricks and PySpark.
  • Build, optimize, and manage Delta Lake tables within Unity Catalog.
  • Develop and support Lakeflow pipelines with incremental processing, CDC, and data quality validations.
  • Integrate enterprise systems, including SharePoint and other external data sources, into the Databricks Lakehouse.
  • Ensure high standards of data quality, governance, security, and operational excellence across data platforms.
  • Monitor, troubleshoot, and optimize data workflows to ensure reliability and performance.
  • Collaborate with business and technical stakeholders to understand requirements and deliver scalable solutions.
  • Build and support AI-powered data solutions, copilots, and intelligent agents leveraging LLMs, RAG frameworks, and vector search technologies.
  • Implement CI/CD best practices and maintain deployment pipelines using Databricks Repos and Asset Bundles.
  • Independently drive deliverables, proactively identify challenges, and resolve issues.

Required Skills & Experience

  • Experience: 6+ years in data engineering roles, with a significant focus on Databricks.
  • Databricks & Data Engineering:
    • Strong hands-on experience with Databricks and modern Lakehouse architecture.
    • Expertise in PySpark, Spark Structured Streaming, SQL, NoSQL databases, Delta Lake, Unity Catalog, Databricks Workflows, and job orchestration.
  • Databricks Lakeflow:
    • Practical experience with Databricks Lakeflow capabilities, including Declarative Pipelines, Lakeflow Connect, change data capture (CDC), data quality validation frameworks, and incremental data processing.
  • AI & Generative AI:
    • Experience designing and building AI agents, copilots, or generative AI solutions using large language models (LLMs), retrieval-augmented generation (RAG), vector search technologies, and prompt engineering.
    • Understanding of integrating enterprise data platforms with AI-driven applications.
  • Data Governance & Security:
    • Strong knowledge of Unity Catalog governance and security practices, including role-based access controls (RBAC), data masking and data protection policies, audit logging, data lineage, and compliance.
  • DevOps & Deployment:
    • Experience with Git-based source control, CI/CD implementation, Databricks Repos, Databricks Asset Bundles (DABs), and automated deployment and release processes.

Preferred Qualifications

  • Experience working with large-scale enterprise data platforms.
  • Strong understanding of modern Lakehouse and Data Mesh principles.
  • Exposure to Azure cloud services and enterprise integration patterns.
  • Excellent problem-solving, communication, and stakeholder management skills.