Senior Data Engineer

Cephas Consultancy Services Private Limited

United States, California, Glendale

Position: One full-time role

Experience: 9–13 years

Location: Glendale, CA (hybrid, 2–4 days onsite)

Visa eligibility: USC/GC/GCEAD/H4EAD/L2

Mandatory skills

  • Databricks experience (primary requirement)
  • Apache Airflow
  • Advanced SQL skills
  • Python
  • Spark/PySpark
  • Scala
  • Experience building and maintaining data pipelines and workflows

Interview process

The process is expected to follow the team’s recent hiring process for similar data engineering positions. It typically consists of an initial screening/vetting round, a technical interview, and a final interview (potentially onsite). The process generally spans 2–3 interview rounds.

Key responsibilities

  • Design, write, test, and deploy data pipelines using PySpark, Scala, SQL, and Python.
  • Meet with stakeholders to gather requirements and translate them into scalable data platform solutions.
  • Use Databricks platform and developer tooling to diagnose errors, audit platform activity, and automate updates across pipelines, objects, and integrations.
  • Explain Spark architecture and pipeline behavior to stakeholders, diagnose root causes, and recommend solutions.
  • Provide solution architecture across AWS, Databricks, Kubernetes, and Airflow (MWAA), including cross-platform integrations.
  • Manage Databricks platform governance, including Unity Catalog, ACLs, lineage, data discovery, and privacy tooling.
  • Build and maintain Kubernetes containers and containerized utilities that support deployed data platform services.
  • Apply networking knowledge to troubleshoot connectivity and integration errors across platform components.
  • Administer the platform by provisioning and removing access, assessing resource utilization, monitoring platform health and cost, and evaluating stakeholder requests.
  • Collaborate with engineers, architects, and product managers to support the Core Data platform, and participate in Agile/Scrum ceremonies.
  • Document platform changes, standards, and pipeline configurations to support data quality and governance.

Qualifications

  • 5+ years of data engineering experience developing and operating large-scale data pipelines.
  • Deep hands-on experience with Databricks and Apache Spark (batch and streaming), including pipeline development in PySpark and/or Scala.
  • Strong understanding of Spark architecture—including executors, stages, partitioning, shuffle, and performance tuning—and the ability to explain trade-offs to technical and non-technical stakeholders.
  • Proficiency with Databricks platform tooling (API, SDK, CLI) for automation, auditing, governance, and operational troubleshooting.
  • Proficiency in SQL, including advanced performance tuning.
  • Hands-on production experience with Airflow (MWAA) for orchestrating data pipelines.
  • Experience managing Databricks platform governance, including ACLs, Unity Catalog, lineage, and access provisioning.
  • Proficiency in Python and at least one additional language (Scala, Kotlin, or SQL-driven pipeline tooling).
  • Experience designing and optimizing scalable ETL/ELT pipelines integrating diverse structured and unstructured data sources.
  • AWS experience, particularly with compute, storage, networking, and IAM; experience with other cloud providers is transferable.
  • Proficiency with Docker and Kubernetes for building and maintaining containerized data platform services.
  • Working knowledge of networking concepts for diagnosing cross-platform integration and connectivity issues.
  • Familiarity with Snowflake and comparable tooling in the Databricks ecosystem.
  • Experience designing and implementing CI/CD and DevOps practices using Git-based workflows.
  • Experience implementing data quality checks, monitoring, and logging for pipeline reliability.
  • Strong analytical and communication skills, a self-starting approach to problem-solving, and willingness to learn new tools and trends.
  • Familiarity with Scrum and Agile methodologies.
  • A bachelor’s degree in Computer Science, Information Systems, or a related field, or equivalent work experience. A master’s degree is a plus.

Preferred

  • Experience with Snowflake.