Databricks data quality: tools, rules and monitoring at scale

Most artificial intelligence and analytics projects fall short of expectations due to poor data quality. This cause is seldom mentioned in failure reports. When the problem is finally identified, it is often mistaken for a model error, a reporting error, or a "let's just re-run it" issue. In reality, the cause is almost always a data problem that began earlier in the process than the point the team examined.

Databricks data quality depends on native tools embedded in the lakehouse: Delta Lake constraints and DLT expectations that block bad data at write time, together with Lakehouse Monitoring for drift and DQX for validating PySpark jobs. It cannot function without an owner assigned to each table in Unity Catalog and with rules recorded as code.

Databricks has devoted the past two years to developing features that address these issues. The platform now offers native tools like Lakehouse Monitoring, Unity Catalog expectations, Delta Live Tables, and DQX. These tools enable users to detect data errors, enforce quality rules, and monitor data drift in large lakehouses. However, these features were built at different times for specific purposes and reside in separate areas of the platform. Significant research is needed to decide which tool is appropriate for a given task.

Shipshape Data assists organisations in preparing their data foundations for AI and advanced analytics. A large part of this effort consists of constructing data quality frameworks specifically on the Databricks platform.

This guide describes the particular features that the platform offers. It includes the rules, the monitoring systems, and the operating model needed to keep these components running in a production environment.

Why data quality behaves differently on Databricks

Most traditional warehouses carry out quality checks solely during the loading process. Data is validated when it enters the system, and after it passes this initial check it is regarded as permanently clean.

Databricks operates in a different way. It employs open storage formats, a distributed compute engine, and a unified governance layer. These capabilities let you verify data quality across the whole process, beginning when the data is first received and extending until it is used in a model.

The lakehouse shifts quality left

Delta Lake sits underneath everything. This architecture lets you enforce constraints, maintain history, and validate data during the write process rather than identifying errors in downstream systems. This approach alters how data is managed. You can detect issues, such as columns containing null values, before the data reaches your silver or gold tables instead of discovering them after a report has been distributed.

The second piece is Unity Catalog, the platform's unified governance layer. In older stacks, quality metadata, lineage, and access controls each resided in separate tools, and none of them communicated with each other. Unity Catalog consolidates all three in a single location, so the rules you write and the monitors you set up remain visible and auditable across the entire platform, whether the workload is a SQL query, a notebook, or an automated pipeline.

The medallion layers decide where each check belongs

The majority of Databricks environments employ a medallion architecture. The structure includes bronze for raw data, silver for cleansed and validated data, and gold for business-ready aggregates. These layers give a clear framework for arranging specific data rules.

• Bronze checks serve a technical purpose. They confirm that the file parses, that the expected fields exist, and that the data types are accurate.
• Silver checks are based on business rules. These rules ensure that values lie within acceptable ranges, that foreign keys resolve, and that unauthorised duplicate entries are identified.
• Gold checks assess aggregate consistency. They confirm that totals are correct and that reported figures align with known benchmarks.

Each layer tolerates incorrect data differently and employs specific tools to detect it. This principle forms the basis of the remainder of this guide.

Why this becomes more important when using AI

These issues are practical rather than academic when building AI or machine learning on a lakehouse, because quality failures compound quickly. If a feature pipeline permits null or out-of-range values to pass, the resulting training data seems correct but gradually degrades the model. This decline often remains unnoticed until performance has already fallen. Databricks integrates MLflow, Feature Store and Delta Lake within a single platform. Consequently, the quality of the data layer directly determines the reliability of your AI outputs.

Set up the correct architecture at the beginning of the project. This costs less than correcting it after a model has generated inaccurate results for three months.

Map your rules to the six dimensions of data quality

Define your goals before writing any constraints. The six dimensions of data quality supply the necessary terminology for this process. Quality cannot be treated as a general concept. This model obliges you to define specific standards for each table. This practice turns business requirements into rules that a pipeline can enforce.

What each dimension catches

• Completeness: are the required fields populated.
• Accuracy: these values must reflect reality when compared to a source of truth or an acceptable range.
• Consistency: does the same fact agree across tables and systems.
• Timeliness: this indicates whether data arrives and updates within your required timeframe.
• Validity: do values match the format, type or allowed set you defined.
• Uniqueness: are there duplicate records inflating counts or breaking joins.

When a quality programme lacks a categorisation step, it frequently generates excessive data. Consequently, many rules lack assigned owners and there is no way to prioritise failures when several alerts appear at the same time.

Where each dimension sits in the medallion layers

Validity and completeness checks are performed at the boundary separating the bronze and silver layers. At this point, raw data is transformed into a reliable format for analysis. These should be expressed as Delta constraints or DLT expectations. Consistency and uniqueness checks typically take place at the silver layer. These checks need table joins or historical data review, which can only be done after the data is structured. Accuracy and timeliness checks continue into the gold layer or Lakehouse Monitoring. Verifying that aggregated numbers align with business benchmarks needs downstream context that is not available in earlier stages.

Start by identifying your most important tables. Determine which data dimensions are most likely to fail, based on their source. Create two or three specific rules for each of these dimensions before beginning any other work. Keep this initial list brief. A small, maintained set of rules is more effective than a comprehensive set that users ignore.

Enforce rules at write time with Delta Lake

Delta Lake offers two built-in methods to stop invalid data from entering a table: table constraints and Delta Live Tables (DLT) expectations. Both methods operate during the write process. If a failure occurs, the system immediately blocks the data to keep it from reaching the underlying storage layers.

Constraints that pertain to the table structure

Delta Lake supports CHECK constraints and NOT NULL constraints at the table level, enforcing them with each write operation. A CHECK constraint is a Boolean expression that applies to one column or a set of columns. For example, a CHECK constraint might require an order status to be one of a specific set of values, or that a customer ID is never null. If a write operation contains a row that does not satisfy a CHECK constraint, the entire transaction fails and no data is written. This gives a firm guarantee: a table with a functioning CHECK constraint will not contain any row that violates it.

Unity Catalog records constraints as table metadata. Users can view these constraints by inspecting the table definition without additional tools. These checks are effective for verifying data validity and completeness at the silver layer, as the data is structured enough for business rules. Constraints are not appropriate for the bronze layer. Raw data is often inconsistent at that stage, and it is preferable to capture failures rather than reject an entire batch of data.

DLT expectations for pipeline-level control

DLT allows you to enforce rules similarly to traditional constraints, but within your declarative pipelines, providing more control over how failures are handled. You define these rules with `expect`, `expect-or-drop`, or `expect-or-fail`, depending on whether you wish to log a warning for a bad row, remove it from the pipeline, or stop the pipeline entirely. This flexibility is the primary advantage over table constraints: you can apply different rules to different sections of the same pipeline, so a minor issue may generate a warning while a critical issue will stop the pipeline.

Handle bad data without breaking your pipelines

If you halt a process for every incorrect row, one of two outcomes will occur. Either people will lower quality standards to keep the system running, or the pipelines will fail frequently, requiring engineers to spend time restarting jobs. Instead, you should decide in advance how to handle each type of data error. This approach keeps the pipeline running while ensuring every failure is recorded for later review.

Quarantine, drop, or fail

The three DLT expectation types correspond to three distinct reactions. Expect records the violation as a measurement and permits the row to continue processing, which is useful when you need to track issues without halting the process. Expect-or-drop removes the row that fails the check from the final output, while the pipeline continues to run. Expect-or-fail stops the entire pipeline, treating the rule as a strict requirement that must always be met. Your choice of which to use depends on how important the field is and whether subsequent steps can operate if a row is missing.

You can apply a quarantine pattern using an expect-or-drop policy. This involves running a parallel stream that directs rejected rows to a specific table for bad records. This method keeps the main pipeline running without interruptions. It also provides a structured location where the data team can investigate errors, resolve source issues, and replay records after they are corrected. We frequently implement this pattern for clients because it creates a recoverable audit trail.

Schema changes that corrupt data quietly

Upstream schema changes commonly cause silent corruption. A new column appears, a type changes, a field gets renamed, and if the pipeline does not handle it explicitly, it either rejects perfectly good data or lets garbage through. Delta Lake's schema enforcement, by default, rejects writes that do not match the current table schema. When you truly need to accommodate new fields, you enable schema evolution with mergeSchema or autoMerge on a per-table basis, ensuring changes remain deliberate.

Using schema enforcement together with documented data contracts in Unity Catalog ensures consistency. Upstream teams can identify breaking changes before they reach the pipeline if they know which fields the contract requires. Unity Catalog provides a versioned schema history that records the timing and nature of each change. This record helps teams trace production incidents back to specific instances of schema drift.

Monitor drift with Lakehouse Monitoring

Write-time rules identify problems as they occur, but they do not reveal whether data that previously passed all checks is declining in quality. Continuous monitoring addresses this by tracking quality metrics across the lakehouse and identifying data drift before it impacts users. Unity Catalog's Lakehouse Monitoring enables this shift from responding to problems after they occur to preventing them, using statistical profiles and anomaly detection features built into the platform.

Setting up a monitor

Lakehouse Monitoring attaches to any Delta table managed by Unity Catalog. Once activated, it runs on a schedule and automatically produces two output tables. The profile table contains statistical summaries per column: null rates, min and max, mean, cardinality. The drift table compares the current window to a baseline and flags any column whose distribution has moved beyond a threshold you set. Both output tables are queryable Delta tables in their own right, allowing you to join them, alert on them, and build dashboards over them with any tool that connects to Databricks SQL.

You can configure a monitor using the Databricks SDK with a few lines of Python code. You must provide the table name, define the granularity window by hour, day, or timestamp column, and select a baseline period. This process automates monitoring for many tables simultaneously. It eliminates the need to deploy a separate observability stack or maintain custom profiling scripts that fail when a schema changes.

Turning metrics into alerts people act on

Unopened metrics stored in a Delta table constitute only storage, not monitoring. Connect Lakehouse Monitoring output to Databricks SQL alerts or to any notification system your team already employs, ensuring that a null rate exceeding a threshold or a column distribution shifting beyond expectations is delivered to a person. Define tolerance bands for every metric in advance. If you do not, the alert channel becomes noisy within two weeks and people begin to ignore it, which nullifies the original purpose.

Treat these monitoring tables as formal data assets. Allocate a documented owner for each table, define an agreed response time, and specify explicit thresholds.

This distinction decides if a programme works effectively or stays a set of unused dashboards.

Use DQX and tests for your PySpark jobs

DQX is an open-source library created for Databricks. It extends the built-in quality tools to cover PySpark transformation jobs. Delta constraints and DLT expectations work well inside declarative pipelines, yet they lack a simple way to validate quality in custom PySpark code. DQX solves this gap: you create rules as Python objects, run them directly in your notebooks and jobs, and keep the checks close to the transformation logic.

What DQX adds on top of native tooling

DQX lets you define a rule set as a list of named checks, each targeting a column and a condition. When you execute these checks on a DataFrame, you obtain a results object that divides passing rows from failing rows. Each failing row is tagged with the rule that flagged it. This simplifies sending incorrect rows to a quarantine path while letting correct rows continue, mirroring the pattern used in DLT expectations but in standard PySpark. DQX also provides severity levels like warning or error, delivering a graded response without needing you to implement this feature in each pipeline.

It also works with Unity Catalog. It records rule results as structured metadata next to the tables the rules validate.

Test the transformation logic

These points do not apply if the underlying transformation code has not been tested.

Use pytest together with a local Spark session to test your PySpark functions before deploying them to production. Create tests for standard operations and for edge cases like null values in required fields, values beyond expected ranges, and unexpected schema structures. Make each test small so that a failure points to the exact function that caused the error.

Place tests in a dedicated directory inside the repository. Set up these tests to run automatically during the continuous integration process for each pull request. This approach catches data transformation errors before they impact the lakehouse. It also gives the team a consistent way to confirm that particular fixes operate as intended.

Build an operating model around the tooling

Tools and rules only hold up if someone owns them. The technical layer, constraints, expectations, monitors, tests, can be as sophisticated as you like, and it still drifts into irrelevance without an operating model behind it. We've watched this happen more than once: a well-built quality framework goes stale within two quarters because nobody was assigned to keep it current, and the alerts start getting muted instead of fixed.

Assign a specific owner to each dataset

Every table in Unity Catalog must have an assigned individual or team. That owner is responsible for defining the table's rules, monitoring its alerts, and correcting any failures within a set timeframe.

If this is not done, alerts build up without being addressed. The rules keep running even after they no longer match business requirements.

You can keep ownership accurate by recording it in Unity Catalog table properties. You should also add quality thresholds and escalation contact information. Save this information inside the data asset. Separate governance documents become outdated quickly, and staff often lose track of where they are.

Manage quality rules as code

You should keep rules, expectations, and monitoring configurations in version control together with the transformation code they govern. If you keep these settings only in a platform UI or an undocumented notebook, they cannot be integrated into CI, reviewed by others, or rolled back reliably. When you store them as code, each change requires a pull request. This creates a clear record of who made the changes and why.

This also simplifies moving rule sets from development to production. You can use the same deployment process you already employ for pipelines. This ensures that a new quality check arrives together with the transformation it covers.

Where to start

Select your most critical tables first. Map them to the six dimensions, then add Delta constraints or DLT expectations at the bronze-to-silver boundary before modifying anything else.

Add Lakehouse Monitoring for drift after the write-time rules are in place, and assign a named owner in Unity Catalog to each dataset from day one. Treat the rules as code from the beginning, and the entire system scales with the platform.

Getting the governance layer right early saves genuine remediation time later, especially when AI workloads sit on the same tables. If you want a second pair of eyes on your Databricks data quality architecture before building further, talk to us. We will tell you plainly where the gaps are and what to fix first.

If your internal reporting no longer meets your needs, you may need our data consulting services. We usually start working with clients at this stage.

Frequently asked questions