About the role
We are hiring a Senior Azure Databricks Data Engineer to design, build and own an enterprise lakehouse on Azure Databricks — from the first workspace to production-grade Bronze, Silver and Gold layers. You have built a Medallion architecture from the ground up, not just worked inside one. You know how to set up the platform so it scales, performs well and stays cost-efficient, with governance, security and data quality built in from day one.
Key responsibilities
Medallion architecture (Bronze / Silver / Gold)
Design and build the Medallion lakehouse from scratch on Azure Databricks and Delta Lake.
Define layer contracts: raw, append-only ingestion in Bronze; cleansed, conformed and deduplicated data in Silver; business-ready, modelled data (star schemas, data marts, aggregates) in Gold.
Build batch and streaming ingestion with Auto Loader, Structured Streaming, Delta Live Tables (Lakeflow Declarative Pipelines) and Azure Data Factory.
Implement incremental processing, CDC, SCD Type 1/2, MERGE patterns, schema evolution and late-arriving data handling.
Set data quality rules and expectations at each layer, with quarantine and alerting for bad records.
Platform architecture and scalability
Architect the Azure Databricks environment: workspaces, dev/test/prod separation, VNet injection, Private Link, ADLS Gen2 storage layout and naming standards.
Design for scale across data volume, number of pipelines, teams and concurrent users.
Define cluster policies, compute types (job, all-purpose, serverless, SQL warehouses) and orchestration with Databricks Workflows.
Build reusable frameworks: metadata-driven ingestion, common libraries, templates and logging.
Performance optimisation
Tune Spark and Delta workloads: partitioning, liquid clustering, Z-ORDER, OPTIMIZE/VACUUM, file sizing, caching, AQE and Photon.
Diagnose and fix skew, shuffle, spill and small-file problems using the Spark UI and query profiles.
Set and meet SLAs for pipeline run time and data freshness.
Cost optimisation (FinOps)
Right-size clusters, use autoscaling, auto-termination, spot instances and serverless where it lowers cost.
Enforce cost controls through cluster policies, tagging and budgets.
Track DBU and Azure spend with system tables and dashboards, and report savings.
Governance, security and standards
Implement Unity Catalog: metastore, catalogs/schemas per layer and domain, external locations, storage credentials.
Set up role-based access, row- and column-level security, data masking and PII handling.
Enable lineage, auditing, data classification and integration with Microsoft Purview.
Manage secrets with Azure Key Vault and identities with Entra ID service principals and managed identities.
Define and document engineering standards: coding guidelines, naming conventions, review process and data contracts.
DevOps and delivery
Build CI/CD with Azure DevOps or GitHub Actions and Databricks Asset Bundles.
Provision infrastructure as code with Terraform or Bicep.
Add unit and integration tests, monitoring and alerting for pipelines.
Leadership
Act as technical lead: mentor engineers, review code and design, and set best practices.
Work with architects, analysts, data scientists and business stakeholders to turn requirements into data products.
Required skills and experience
7+ years in data engineering, including 3+ years building production workloads on Azure Databricks.
Proven track record of building a Medallion architecture from the ground up — be ready to walk through one you designed end to end.
Expert in PySpark and Spark SQL; strong Python and SQL.
Deep knowledge of Delta Lake: ACID transactions, time travel, MERGE, CDF, OPTIMIZE, liquid clustering.
Hands-on with Unity Catalog and data governance: access control, lineage, auditing, PII protection.
Strong Azure stack: ADLS Gen2, Azure Data Factory, Key Vault, Entra ID, Event Hubs, Azure Monitor, networking basics (VNet, Private Endpoints).
Proven results in Spark performance tuning and cloud cost optimisation, with numbers to show for it.
Data modelling: dimensional modelling (Kimball), Data Vault awareness, SCD patterns.
CI/CD and IaC: Git, Azure DevOps or GitHub Actions, Databricks Asset Bundles, Terraform.
Clear communicator who can explain design trade-offs to both engineers and business stakeholders.
Nice to have
Certifications: Databricks Certified Data Engineer Professional, Microsoft Certified: Azure Data Engineer Associate (DP-203) or Fabric Data Engineer (DP-700), Databricks Lakehouse Platform Architect.
Experience migrating from on-prem (SQL Server, Hadoop, Informatica, SSIS) or Synapse to Databricks.
Databricks SQL, Lakehouse Monitoring, MLflow or Feature Store exposure.
dbt on Databricks, Great Expectations, or similar data quality tools.
Power BI integration and semantic model design on Gold data.
Experience with data mesh or domain-oriented data products.
