Great Expectations data quality: how to automate validation

A dashboard showed wrong data for three days before the error was spotted. Eventually, someone raised a question about a drop in revenue. The investigation revealed that the cause originated two stages earlier in the pipeline. A batch of corrupted records was missed because there was no monitoring for that particular issue.

Great Expectations data quality software turns your dataset assumptions into automated tests. These rules, known as expectations, are grouped into suites. A checkpoint executes these suites on real data each time new data arrives. This approach substitutes manual spot checks and outdated wiki documentation with one resource that acts as both a test and a plain-language description of clean data.

Manual spot checks catch the obvious stuff.

Users frequently miss errors such as negative age values or improperly formatted email addresses. Great Expectations is an open-source Python library built to detect these issues. It turns your data assumptions into automated tests that execute whenever new data is processed. The tool gives specific details about what failed and where the error happened.

At Shipshape Data, we often construct pipelines for clients who have faced this particular issue. Most clients have run into the problem multiple times before getting in touch to us. The sequence of events remains consistent. A model or report yields incorrect results, and an employee spends an entire day tracing the error back to a source table. After the specific column is identified, the fix takes only ten minutes.

Great Expectations won't stop bad data arriving.

This process cuts a full day of tracing down to ten minutes. The validation pinpoints the exact columns and rows for you.

The library is extensive, yet the mechanics are simple. You write expectations, which are statements that define how a column or table should look. For instance, you can state that a column must never have null values, must be between 0 and 120, or must match one of three particular values. You group these expectations into a suite and apply the suite to a data batch via a checkpoint. The system then reports which expectations passed or failed and gives the specific margins of failure. If you run the same checkpoint on new data the next day, the process runs automatically without opening a notebook.

This guide describes the complete setup process. It outlines how to install the package, link it to a data source, create a permanent test suite, and integrate validation into a production checkpoint. Each individual step is simple. The main challenge for most teams is making sure these four components work together and stay functional as data changes.

Why Great Expectations became the default for Python pipelines

Data quality tools existed before Great Expectations. Most of those tools required writing tests in one place and keeping documentation in another. This approach depended on people manually updating both files each time the data model changed. In practice, this seldom occurred. Wikis quickly became outdated, validation scripts were ignored, and the documentation and code grew inconsistent over time. Great Expectations solved this problem by merging the test and the documentation into one file. When you write an expectation, you produce both a technical rule for the data and a description readable by a non-technical person. This capability is why the tool became the standard for data quality checks in Python pipelines.

Documentation that cannot drift from reality

Data documentation usually resides outside the data pipeline-in a wiki, a spreadsheet, or an outdated presentation. This documentation becomes inaccurate as soon as a column changes or a table is renamed, and the issue often remains unnoticed until an analyst uses it and hits an error. When you declare that `customer_id` must never be null, you are at the same time creating a test and a documentation record: this column must never contain nulls.

Data stays synchronised because there is only a single copy.

Each suite compiles into a collection of static pages named Data Docs. Stakeholders can read these pages without needing Python. The pages show the current expectations, the results of the latest run, and profiling statistics for the data. A finance lead can open a page to view the specific definition of clean data for the customer table. This information is written in plain language and generated from the same code used for the checks. Documentation and validation stay integrated because they belong to the same process.

One suite, many backends

The library executes the same expectation code on Pandas dataframes, PostgreSQL, Snowflake, BigQuery, and Spark, so a suite created for a CSV file during development works unchanged when you point it at a warehouse table in production. That matters more than it sounds: teams switch platforms, add a second warehouse, or begin validating a dataset earlier in the pipeline than before, and the expectation logic does not need to be rewritten each time. The same suite also covers batch loads and streaming updates. You define the rule once and apply it wherever the data resides that week.

Install the package and start a data context

Great Expectations depends on a data context. This folder holds your expectation suites, checkpoints, and configuration files.

This is the first thing you create because nothing else works without it.

Install via pip

You need Python 3.8 or newer. Installing the core package is a single command in your terminal, pip install great_expectations, and it includes everything needed to validate Pandas dataframes out of the box. If your data resides in a warehouse, install the matching extra for your backend: Snowflake, BigQuery, PostgreSQL, and Spark are all supported via optional extras like great_expectations[snowflake]. You can chain several inside the same brackets if you need more than one backend, which most teams eventually do once they begin validating data at multiple pipeline stages.

Initialise your project

Run the command great_expectations init from the root of your project. This command creates a great_expectations folder. The setup wizard will ask whether you prefer local file storage or a cloud backend for Data Docs. You should choose local storage first to confirm the system works end to end. You can switch to cloud storage after verifying the rest of the configuration. The new folder contains:

• A main configuration file recording your connections and settings
• A directory for the expectation suites you write
• A directory for the checkpoints that run those suites against data
• A plugins directory for any custom expectations or actions
• An uncommitted directory for validation results and generated docs

The uncommitted directory is worth noting: it changes on each validation run, so it belongs in your .gitignore, not in your repository.

Direct the tool towards existing data

A data context does not function until you specify the location of your data.

Great Expectations calls this a datasource. You must set up a datasource before you can create a suite or validate actual data rows.

Configure a datasource

Execute great_expectations datasource new to begin the interactive setup. The tool prompts you to pick a backend: select pandas for CSV or Excel files, SQL for databases, or Spark for distributed data. When you choose SQL, you need to supply a connection string, for example a Postgres URL that includes your host, port, and database name. If you intend to automate your processes or operate in multiple environments, avoid using the wizard. Instead, modify the configuration file to include a datasource block. You need to define the execution engine and a data connector. Direct the connector to a particular directory or a set of tables, and Great Expectations will detect it when the context loads. Most teams adopt this approach after finishing a proof of concept because the configuration can be kept in version control and deployed.

Check the connection before you go further

Load the context in a Python shell, list the configured datasources, and confirm the one you just set up shows up in the result. Then pull an actual batch: build a batch request naming the datasource, the data connector and the table or file, hand it to the context to get a validator, and print the first few rows. If you see rows, the connection works. If you see a credentials error or a missing-table error, you get told exactly what failed rather than a generic timeout, which saves a genuinely annoying afternoon compared with debugging a silent connection issue three steps later.

Write an expectation suite that says something true

This step is the most crucial, yet teams frequently finish it too fast. An expectation suite consists of a set of rules applied to a particular dataset. You may generate an empty suite via the command line. Once you have chosen your datasource and asset, Great Expectations lets you draft a suite by profiling a data sample or by manually writing the rules. For critical tasks, you ought to select the manual option. The interactive profiler detects statistical patterns, yet it does not comprehend specific business rules. For instance, it will not recognise that the account_status field must be restricted to active, inactive, or suspended. If a fourth value appears in the sample, the profiler will incorporate it into the rules.

Start with checks that would catch a real incident

Avoid writing fifty expectations on the first day. Begin with checks that match an incident you have experienced or would actually notice if it happened.

• The customer_id field must not have null values. A null ID stops all downstream joins from working correctly.
• Email addresses must adhere to a standard format. If an address is formatted incorrectly, the system will be unable to send notifications. These errors frequently stay unnoticed until a customer reports the issue.
• The age value must lie between 0 and 120. An entry like minus four or four thousand shows that a failure happened in an earlier part of the data process.
• The account_status column must use a limited set of predefined values. If a typo appears in this column, the affected rows will be left out of dashboard filters that depend on the standard list of statuses.

These individual checks are not significant by themselves. However, combining them lets you spot data problems via validation reports on the day they happen, instead of discovering them later from customer complaints. You should add schema checks to confirm that expected columns and data types exist before adding checks for distributions and statistics. Schema drift makes pipelines fail completely. Conversely, a shifted distribution usually needs investigation but does not force you to stop the process.

Save the suite, and include any tests that fail

Turn off the option that discards failed expectations when you save them. Most users make this mistake on their first attempt. By default, the system removes an expectation when the current data sample violates it. You should modify this setting because a violation often indicates a real data quality issue. Even if the data has not yet passed, the expectation stays a valid rule for your project. Recording the rule as a documented failure is more effective than letting the system delete it.

Every expectation you document becomes a requirement that your data must meet later, regardless of whether the current batch of data satisfies that condition.

Run validation through checkpoints, not one-off scripts

A checkpoint brings together an expectation suite, a particular batch request, and a set of actions for the results. This lets you validate a table using a single command rather than recalling a five-step sequence. Checkpoints are used to schedule tasks, trigger actions from a CI pipeline, or invoke calls from Airflow, Prefect, or Dagster. Although suites are collections of rules kept in files, a checkpoint is needed to apply those rules to real data.

Building the checkpoint

A checkpoint can be created from the command line and assigned a name. The setup process will walk you through choosing a datasource, a suite, and the actions to perform after validation, for example updating Data Docs and storing the results. For any checkpoints you plan to deploy, the configuration should be written in code. Place this code under version control together with the corresponding pipeline, as you do with database migrations and schema changes.

What to do when a checkpoint fails

The checkpoint provides a success flag together with a detailed report that lists the expectations that failed, the exact rows affected, and the magnitude of the failure. How your pipeline processes this flag matters more than the validation logic itself. If a core table has a null customer_id, you should halt the pipeline. Conversely, when a known legacy source yields a handful of ages just beyond the expected range, you should record the event and keep the pipeline running. Before configuring alerts, you need to differentiate between these situations. Failing to do so will probably cause you to dismiss alerts as noise, or to disturb staff over insignificant data problems.

Read the results without drowning in noise

Great Expectations generates a static site called Data Docs for each validation run, displaying which expectations passed, which failed, sample values that violated a rule, and profiling statistics for the underlying data. Create the docs and open the local site to view them. Place it on shared storage so the entire team, even those who did not write the suite, can view the current data quality status without needing to ask anyone.

Data Docs is too slow for urgent issues since users do not visit the site hourly. Link checkpoint actions to Slack, Microsoft Teams, PagerDuty, or email. This makes failures appear in the tools where your team already works. Appropriate thresholds must be set. A soft check that falls two per cent below expectations can be logged without triggering an alert. A hard schema violation or a null-check failure on a primary key should cause an immediate page. Setting these thresholds incorrectly will cause the tool to be dismissed as background noise or distrusted as an alarm that sounds too frequently. In either case, the tool will not fulfil its purpose.

Where this goes wrong in practice

The most common mistake is not technical. Teams frequently generate forty expectations in the first week to cover every column. If the data model changes three months later, nobody updates those expectations. An outdated test suite that flags harmless changes as failures causes the team to disregard the tool. That result is worse than having no tool. Having no safety net is better than a false sense of security. Start with ten checks that reflect real incidents. When you demonstrate that these checks work, you may add more.

Viewing validation as a project with a set end date is wrong. Validation needs to be a regular practice that handles schema changes. Changes happen when a column is added, a business rule is changed, or an upstream vendor modifies a date format. A validation suite correct in January will be wrong by June if nobody owns it. Assign an owner to the suite the same way you would for a dashboard. Review the suite each time the underlying table changes instead of using a fixed calendar that ignores data movements.

Teams also skip the datasource step mentioned earlier and embed connection details straight into notebooks.

This approach works for a demonstration. It breaks when two users try to run the same checkpoint against different environments. In such cases, you will waste time fixing credential errors rather than spotting data quality problems.

Integrating this into the broader architecture

Great Expectations offers a way to validate data. It does not solve the underlying architectural problems that let poor data enter a system, nor does it handle the effects of unmonitored columns that start to drift. It also lacks a method for spotting similar problems in unstructured data that has no formal validation schema. We build these systems for clients because tools are constrained by pipeline design quality. A checkpoint in a badly designed pipeline will keep flagging the same error repeatedly if the root cause remains unaddressed.

If you already use Great Expectations and it is silently filled with noisy, unmaintained checks, or if you are figuring out where validation belongs in your pipeline, talk to us. We prefer to help you fix the three important checks rather than watch you create another forty that no one reviews.

Our data consultants build these systems for UK organisations. We focus on the foundation to ensure reliable results.

Frequently asked questions