Poor data quality is the reason most AI and analytics projects quietly under-deliver. Nobody puts it on the failure report, because by the time anyone notices, the problem looks like a model issue, a reporting issue, or a "let's just re-run it" issue. It was almost always a data issue, and it started upstream of where anyone was looking.
Databricks has spent the last couple of years closing that gap. Lakehouse Monitoring, Unity Catalog expectations, Delta Live Tables, DQX: there is now a genuine set of native tools for catching bad data early, enforcing rules, and watching for drift across lakehouses running to petabytes. The trouble is that these tools sit in different corners of the platform, built at different times for different jobs, and working out which one solves which problem takes some digging.
We help organisations at Shipshape Data get their data foundations ready for AI and advanced analytics, and a good chunk of that work is building data quality frameworks on Databricks specifically. This guide walks through what the platform actually gives you: the rules, the monitoring, and the operating model that has to sit around both if any of it is going to hold up in production.
Why data quality behaves differently on Databricks
Most traditional warehouses check quality once, at load. Data gets validated on the way in, and once it passes, it is treated as clean for good. Databricks does not work that way. Because it combines open storage formats, a distributed compute engine and a single governance layer, you can check quality at every stage of the data's life, from the moment it lands to the moment it feeds a model.
The lakehouse shifts quality left
Delta Lake sits underneath everything as a transactional storage layer, which means you can enforce constraints, keep history, and validate data at the point of write rather than waiting for it to break something downstream. That is a real shift. You stop discovering a column full of nulls after a report has already gone out wrong, and start catching it before the data even settles into your silver or gold tables.
The second piece is Unity Catalog, the platform's unified governance layer. In older stacks, quality metadata, lineage and access controls each lived in their own tool, and none of them talked to each other. Unity Catalog puts all three in one place, so the rules you write and the monitors you set up stay visible and auditable across the whole platform, whether the workload is a SQL query, a notebook, or an automated pipeline.
The medallion layers decide where each check belongs
Most Databricks environments run on a medallion architecture: bronze for raw data, silver for cleansed and validated data, gold for business-ready aggregates. That layering gives you a natural map for where different rules belong.
- Bronze checks are technical: does the file parse, are the expected fields present, are the data types right.
- Silver checks are business rules: are values within an acceptable range, do foreign keys resolve, are there duplicates that should not be there.
- Gold checks are aggregate consistency: do totals add up, do the reported figures match known benchmarks.
Each layer has a different tolerance for bad data and a different tool suited to catching it, which is the theme running through the rest of this guide.
Why it matters more once AI is involved
None of this is academic once you are building AI or machine learning on top of the lakehouse, because quality failures compound fast there. A feature pipeline that quietly lets nulls or out-of-range values through produces training data that looks fine on the surface and degrades the model slowly, in a way nobody notices until performance has already slipped. Databricks ties MLflow, Feature Store and Delta Lake together in one platform, so the quality of your data layer shows up directly in how reliable your AI outputs are.
Get the architecture right first. It is cheaper than fixing it after a model has been quietly wrong for three months.
Map your rules to the six dimensions of data quality
Before you write a single constraint, decide what you are actually trying to protect. The six dimensions of data quality give you the vocabulary for that. Instead of treating quality as a vague good thing to have more of, the six-dimension model forces you to say exactly what good data looks like for a given table, which turns a business requirement into something a pipeline can actually enforce.
What each dimension actually catches
- Completeness: are the required fields populated.
- Accuracy: do values reflect reality, checked against a source of truth or an acceptable range.
- Consistency: does the same fact agree across tables and systems.
- Timeliness: does data arrive and update inside the window you need it to.
- Validity: do values match the format, type or allowed set you defined.
- Uniqueness: are there duplicate records inflating counts or breaking joins.
Skip the categorisation step and a quality programme tends to generate noise rather than insight: dozens of rules, no clear owner for any of them, and no way to tell which failures actually matter when three fire at once.
Where each dimension sits in the medallion layers
Validity and completeness checks belong at the bronze-to-silver boundary, where raw, messy data is turned into something reliable enough to analyse. These are the ones you write as Delta constraints or DLT expectations, covered below. Consistency and uniqueness tend to sit at the silver layer, because checking them usually means joining across tables or looking at history, which you cannot do before the data has structure. Accuracy and timeliness stretch into the gold layer or into Lakehouse Monitoring, because confirming an aggregated number is accurate against a business benchmark needs downstream context you do not have earlier.
A good way to start: take your most important tables, work out which dimension is most likely to fail given where the data comes from, and write two or three concrete rules per dimension before you build anything else. Keep the list short at first. A focused set of rules you actually maintain beats a comprehensive one nobody looks at.
Enforce rules at write time with Delta Lake
Delta Lake gives you two native ways to stop bad data before it settles into a table: table constraints, and Delta Live Tables (DLT) expectations. Both act at write time, so a failure stops the data at the door instead of letting it drift quietly into the layers below.
Constraints on the table itself
Delta Lake supports CHECK constraints and NOT NULL constraints at the table level, enforced on every write. A CHECK constraint is a Boolean expression tied to a column or set of columns, for example requiring an order status to be one of a fixed list of values, or a customer ID to never be null. If any row in a write breaks that expression, the whole transaction fails and nothing gets written. That is a hard guarantee: a table with a working CHECK constraint will never contain a row that violates it.
Constraints are stored as table metadata in Unity Catalog, so anyone inspecting the table definition can see them without extra tooling. They suit validity and completeness checks well at the silver layer, where the data is structured enough for business rules to make sense. They are the wrong tool at bronze, where raw data is expected to be messy and you want to capture the failures rather than reject the whole batch.
DLT expectations for pipeline-level control
DLT expectations give you the same rule-based enforcement inside declarative pipelines, with more say over what happens when a row fails. You define one with expect, expect-or-drop, or expect-or-fail, depending on whether a bad row should be warned about, filtered out, or stop the pipeline outright. That flexibility is the real advantage over table constraints: different rules inside the same pipeline can be treated differently, so a minor field can just log a warning while a critical one halts the run.
Handle bad data without breaking your pipelines
Fail hard on every bad row and you get one of two outcomes: thresholds get quietly lowered to keep things running, or pipelines break constantly and engineers spend their week restarting jobs instead of building anything. The fix is to decide, upfront, how each kind of bad data should be handled, so the pipeline stays healthy while every failure still gets captured for someone to look at.
Quarantine, drop, or fail
The three DLT expectation types map to three responses. Expect logs the violation as a metric and lets the row through, useful when you want visibility without enforcement. Expect-or-drop filters the bad row out of the output but keeps the pipeline running. Expect-or-fail halts the whole thing, treating the rule as a hard contract that must never be broken. Which one you pick depends on how critical the field is and whether anything downstream can tolerate a missing row at all.
A quarantine pattern pairs well with expect-or-drop: run a parallel stream that captures rejected rows and lands them in a dedicated bad-records table, so the main pipeline stays unblocked and your data team has a structured place to investigate, fix the source issue, and replay corrected records once they are clean. Of the patterns we implement for clients, this is the one that comes up most often, because it gives you a recoverable audit trail instead of a silently shrinking dataset.
Schema changes that corrupt data quietly
Upstream schema changes are a common source of silent corruption. A new column turns up, a type changes, a field gets renamed, and if the pipeline does not handle it explicitly, it either rejects perfectly good data or lets garbage through. Delta Lake's schema enforcement rejects writes that do not match the current table schema by default. When you genuinely need to accommodate new fields, you turn on schema evolution with mergeSchema or autoMerge on a per-table basis, so changes stay deliberate rather than automatic.
Pairing schema enforcement with documented data contracts in Unity Catalog closes the loop. When upstream teams know which fields are contractually required, they can flag breaking changes before they land in your pipeline rather than after. Versioned schema history in Unity Catalog gives you a clean record of what changed and when, which matters when you are tracing a production incident back to a specific piece of schema drift.
Monitor drift with Lakehouse Monitoring
Write-time rules catch problems as they arrive, but they will not tell you whether data that passed every check last month is quietly getting worse. That is a job for continuous monitoring: tracking quality metrics across the lakehouse and flagging drift before it reaches anyone downstream. Unity Catalog's Lakehouse Monitoring is where this moves from reactive to proactive, statistical profiles and anomaly detection built into the platform rather than bolted on from outside.
Setting up a monitor
Lakehouse Monitoring attaches to any Delta table managed by Unity Catalog. Once it is on, it runs on a schedule and produces two output tables automatically. The profile table holds statistical summaries per column: null rates, min and max, mean, cardinality. The drift table compares the current window against a baseline and flags any column whose distribution has moved past a threshold you set. Both output tables are queryable Delta tables in their own right, so you can join them, alert on them, and build dashboards over them with anything that connects to Databricks SQL.
You configure a monitor in a few lines of Python through the Databricks SDK: name the table, set the granularity window (hourly, daily, or by a timestamp column), and pick a baseline period. That is enough to automate monitoring across dozens of tables without standing up a separate observability stack or maintaining custom profiling scripts that go stale the moment someone changes a schema.
Turning metrics into alerts people actually act on
Metrics sitting in a Delta table nobody looks at are not monitoring, they are storage. Wire Lakehouse Monitoring output into Databricks SQL alerts or whatever notification system your team already uses, so a null rate crossing a threshold or a column distribution moving further than expected actually reaches someone. Set tolerance bands for each metric upfront rather than alerting on every small wobble. Otherwise the alert channel fills with noise within a fortnight and everyone starts ignoring it, which defeats the point of building it in the first place.
Treat these monitoring tables as proper data assets: give them a documented owner, an agreed response time, and explicit thresholds. That is what separates a quality programme that actually functions from a folder of dashboards nobody opens.
Use DQX and tests for your PySpark jobs
DQX is an open-source library built specifically for Databricks that extends native quality tooling to cover PySpark transformation jobs. Delta constraints and DLT expectations work well inside declarative pipelines, but they do not give you a clean way to validate quality inside custom PySpark code. DQX fills that gap: define rules as Python objects, apply them directly inside your notebooks and jobs, and keep the checks close to the transformation logic instead of shunting them off into a separate layer.
What DQX adds on top of native tooling
DQX lets you define a rule set as a list of named checks, each targeting a column and a condition. Run those checks against a DataFrame and you get back a results object splitting passing rows from failing ones, each failure tagged with the rule that caught it. That makes it straightforward to route bad rows into a quarantine path and let clean rows continue, the same pattern you get from DLT expectations, just inside plain PySpark. DQX also supports severity levels, warning versus error, so you get the same graded response without rebuilding it yourself inside every pipeline.
It integrates with Unity Catalog too, logging rule results as structured metadata next to the tables they validate rather than printing them to a notebook output nobody reopens.
Test the transformation logic itself
None of this matters if the transformation code underneath is untested. Use pytest with a local Spark session to write unit tests against your PySpark functions before they ever run in production. Cover the happy path and the edge cases: nulls in required fields, out-of-range values, schema shapes you did not expect. Keep each test small so a failure points straight at the function responsible instead of somewhere inside four hundred lines of notebook.
Put tests in their own directory in the repository and run them on every pull request as part of CI. It catches transformation bugs before they reach the lakehouse, and gives the team a repeatable way to check that a fix has actually fixed something rather than just moved the symptom somewhere else.
Build an operating model around the tooling
Tools and rules only hold up if someone owns them. The technical layer, constraints, expectations, monitors, tests, can be as sophisticated as you like, and it still drifts into irrelevance without an operating model behind it. We have watched this happen more than once: a well-built quality framework goes stale within two quarters because nobody was assigned to keep it current, and the alerts start getting muted instead of fixed.
Give every dataset a named owner
Every table in Unity Catalog needs a person or team responsible for defining its rules, reviewing its alerts, and resolving failures inside an agreed time. Without that, alerts pile up unacknowledged and rules keep running long after they stopped reflecting what the business actually needs.
The practical way to make ownership stick is to record it directly in Unity Catalog table properties, alongside the quality thresholds and who to escalate to if the primary owner is out. Keep that information inside the data asset itself. A separate governance document goes out of date within a few months, and nobody remembers where it lives.
Treat quality rules as code
Rules, expectations and monitoring configurations belong in version control next to the transformation code they protect. Leave them living only in a platform UI or an undocumented notebook and they become invisible to CI, impossible to review, and hard to roll back cleanly. Store them as code and every change goes through a pull request, with a clear record of who changed what and why.
This also makes it far easier to promote rule sets from development through to production using the same deployment process you already use for pipelines, so a new quality check arrives alongside the transformation it protects rather than getting bolted on after something has already broken and cost someone a Friday afternoon.
Where to start
Pick your most critical tables first. Map them against the six dimensions, then add Delta constraints or DLT expectations at the bronze-to-silver boundary before you touch anything else. Layer in Lakehouse Monitoring for drift once the write-time rules are in place, and give every dataset a named owner in Unity Catalog from day one rather than retrofitting ownership after the first missed alert. Treat the rules as code from the start, and the whole thing scales with the platform instead of falling behind it.
Getting the governance layer right early saves real remediation time later, especially once AI workloads sit on top of the same tables. If you want a second pair of eyes on your Databricks data quality architecture before you build any further, talk to us. We will tell you plainly where the gaps are and what to fix first.