A dashboard goes quiet for three days before anyone notices the numbers are wrong. Someone finally asks why revenue dipped, you pull the thread, and the answer sits two pipeline stages back: a batch of corrupt records nobody was watching for.
Manual spot checks catch the obvious stuff. They miss the row where a customer's age reads minus four, or the batch where half the email addresses stopped matching a sensible pattern. Great Expectations exists to close that gap. It is an open source Python library that turns your assumptions about a dataset into tests that run automatically, every time new data lands, and it tells you exactly what broke and where.
We spend a lot of our time at Shipshape Data building pipelines for clients who have been burned by this exact problem, usually more than once before they call us. The pattern is always the same: a model or a report goes wrong, someone spends a day tracing it back to a source table, and the fix takes ten minutes once you know which column to look at. Great Expectations will not stop bad data arriving. What it does is shrink that day of tracing down to the ten minutes, because the validation already told you which column and which rows.
The mechanics are simple even though the library is not small. You write expectations, statements about what a column or table should look like: never null, always between 0 and 120, always one of three allowed values. You group expectations into a suite, point the suite at a batch of data through something called a checkpoint, and Great Expectations tells you which expectations passed, which failed, and by how much. Run the same checkpoint again tomorrow against tomorrow's data and you get the same answer, automatically, without anyone opening a notebook.
This guide walks through the setup end to end: installing the package, connecting it to a real data source, writing a suite worth keeping, and wiring validation into a checkpoint you can trust in production. None of it is complicated on its own. Getting all four pieces working together, and keeping them working as your data changes, is where most teams either succeed or quietly give up.
Why Great Expectations became the default for Python pipelines
Data quality tooling existed long before Great Expectations. Most of it asked you to write tests in one place, keep documentation in another, and trust that someone remembered to update both when the data model changed. In practice nobody did. Wikis went stale within a month, validation scripts sat in a folder nobody opened, and the two versions of the truth drifted further apart every sprint. Great Expectations closed that gap by making the test and the documentation the same artefact: write an expectation and you have written a rule your data must follow and a sentence a non-technical reader can understand, at the same time. That is the whole reason it became the standard for data quality checks in Python pipelines rather than one option among many.
Documentation that cannot drift from reality
Traditional data documentation lives somewhere separate from the pipeline: a wiki page, a spreadsheet, a slide from eighteen months ago nobody has touched since. It goes stale the moment someone changes a column or renames a table, and nobody notices until a new analyst trusts it and gets burned. Write the expectation that customer_id must never be null and you have created a test and a piece of documentation in the same line: this column must never contain nulls, full stop. There is no second copy to fall out of sync, because there is only one copy.
Every suite compiles into a set of static pages, Great Expectations calls this Data Docs, that a stakeholder can read without touching Python. The pages show which expectations exist, what passed and failed on the last run, and profiling statistics about the underlying data. A finance lead can open that page and see exactly what "clean" means for the customer table, in plain language, generated from the same code that actually runs the checks. Documentation and validation stay together because they were never two separate things to begin with.
One suite, many backends
The library runs the same expectation code against Pandas dataframes, PostgreSQL, Snowflake, BigQuery and Spark, so a suite written against a CSV file during development works unchanged once you point it at a warehouse table in production. That matters more than it sounds: teams migrate platforms, add a second warehouse, or start validating a dataset earlier in the pipeline than they used to, and the expectation logic does not need rewriting each time. The same suite covers batch loads and streaming updates too. You define the rule once and apply it wherever the data happens to be that week.
Install the package and start a data context
Everything in Great Expectations hangs off something it calls a data context: the folder that holds your expectation suites, your checkpoints, and your configuration. Nothing else works until this exists, so it is the first thing you build.
Install via pip
You need Python 3.8 or later. Installing the core package is a single command in your terminal, pip install great_expectations, and it brings in everything required to validate Pandas dataframes out of the box. If your data lives in a warehouse rather than a dataframe, install the matching extra for your backend: Snowflake, BigQuery, PostgreSQL and Spark are all supported through optional extras such as great_expectations[snowflake]. You can chain several inside the same brackets if you need more than one backend, which most teams eventually do once they start validating data at more than one pipeline stage.
Initialise your project
From your project root, run great_expectations init. That single command creates a great_expectations folder, and the wizard asks whether you want local file storage or a cloud backend for Data Docs. Choose local storage first, get something working end to end, and move to cloud storage once the rest of the setup is proven. The folder it creates holds:
- A main configuration file recording your connections and settings
- A directory for the expectation suites you write
- A directory for the checkpoints that run those suites against data
- A plugins directory for any custom expectations or actions
- An uncommitted directory for validation results and generated docs
The uncommitted directory is worth remembering: it changes on every validation run, so it belongs in your .gitignore, not your repository.
Point it at data that actually exists
A data context does nothing until it knows where your data lives. Great Expectations calls this a datasource, and you configure one before you can create a suite or run a validation against real rows rather than a demo file.
Configure a datasource
Run great_expectations datasource new for an interactive setup, and the wizard asks what kind of backend you are pointing at: pandas for CSV or Excel files, SQL for a database, Spark for distributed data. Choose SQL for a warehouse connection and it asks for a connection string, something like a Postgres URL with your host, port and database name filled in. For anything you want to automate or run in more than one environment, skip the wizard and edit the configuration file directly instead: add a datasource block naming the execution engine and a data connector, point the connector at a directory or a set of tables, and Great Expectations picks it up the next time it loads the context. This is the version most teams end up using once they move past a first proof of concept, because it is the version you can put in version control and deploy.
Check the connection before you go further
Load the context in a Python shell, list the configured datasources, and confirm the one you just set up shows up in the result. Then pull an actual batch: build a batch request naming the datasource, the data connector and the table or file, hand it to the context to get a validator, and print the first few rows. If you see rows, the connection works. If you see a credentials error or a missing-table error, you get told exactly what failed rather than a generic timeout, which saves a genuinely annoying afternoon compared with debugging a silent connection issue three steps later.
Write an expectation suite that says something true
This is the part that actually matters, and it is also the part teams rush. An expectation suite is a named collection of rules for one dataset. Create an empty one through the command line, choose your datasource and asset, and Great Expectations offers to build the first draft interactively by profiling a sample of your data, or leave it to you to write manually. Take the manual route for anything that matters. The interactive profiler is good at spotting statistical patterns. It has no idea that your business rule says account_status can only ever be active, inactive or suspended, and it will happily accept a fourth value it happened to see in the sample.
Start with checks that would actually catch a real incident
Resist the urge to write fifty expectations on day one. Start with the checks that map to an incident you have actually had, or would genuinely notice if it happened:
- Customer_id should never be null, because a null id breaks every downstream join
- Email should match a sensible pattern, because a malformed address means a failed notification nobody catches until a customer complains
- Age should sit between 0 and 120, because a data entry error of minus four or four thousand tells you something upstream broke
- Account_status should only ever be one of a small fixed set of values, because a typo in that column silently drops rows out of every dashboard filter that assumes the clean list
Each of those is boring on its own. Together they are the difference between finding out about a data problem from a customer complaint and finding out about it from a validation report the same morning it happened. Add checks for schema, the columns you expect in roughly the types you expect, before you add checks for distributions and statistics. Schema drift breaks pipelines outright, whereas a shifted distribution is usually something to investigate rather than something to halt on.
Save the suite, including the ones that fail
Save your expectations with the option to discard failed expectations switched off, not on. This is the detail almost everyone gets wrong the first time. If you write an expectation and your current sample violates it, the default behaviour throws that expectation away rather than saving a documented failure. You usually want the opposite: you have just discovered a real data quality problem, and the expectation you wrote to describe it is worth keeping even though today's data does not pass it yet. Saving it as a known, tracked failure is more useful than pretending the rule never existed.
Every expectation you save is a contract your data has to honour from now on, whether or not today's batch happens to keep it.
Run validation through checkpoints, not one-off scripts
A checkpoint bundles an expectation suite with a specific batch request and a set of actions to take on the result, so validating a table becomes one command instead of five steps you have to remember to run in the right order. Checkpoints are what you actually schedule, trigger from a CI pipeline, or call from Airflow, Prefect or Dagster. Suites on their own are just rules sitting in a file until a checkpoint puts them to work against real data.
Building the checkpoint
Generate one through the command line, name it, and the wizard walks you through picking a datasource, a suite, and the actions to run once validation finishes, typically updating Data Docs and storing the result. For anything you plan to deploy, build the checkpoint configuration in code instead and keep it in version control alongside the pipeline it protects, the same way you would keep a database migration next to the schema it changes.
What a failed checkpoint should actually trigger
Running the checkpoint returns a result with a single success flag plus the full detail behind it: which expectations failed, on which rows, by how much. What your pipeline does with that flag is the design decision that matters more than the validation logic itself. A null customer_id on a core table is worth stopping the pipeline for. A handful of ages sitting just outside your expected range from a known legacy source is often worth logging and moving on. Decide which is which before you wire alerting up, or you will spend the first month either ignoring every alert because most of them are noise, or paging someone at two in the morning for a data point nobody actually cares about.
Read the results without drowning in noise
Great Expectations builds a static site, Data Docs, out of every validation run: which expectations passed, which failed, sample values that broke a rule, and profiling statistics for the underlying data. Build the docs and open the local site to see it. Host it on shared storage and the whole team, not just whoever wrote the suite, can check current data quality status without asking anyone.
For anything urgent, Data Docs alone is too slow, because nobody checks a website every hour. Wire checkpoint actions into Slack, Microsoft Teams, PagerDuty or email and failures land where your team already looks. Set thresholds rather than alerting on every single failure: a validation that sits two percent below expectation on a soft check can log quietly, while a hard schema break or a null-check failure on a primary key should page someone straight away. Get that threshold wrong in either direction and the tool becomes background noise everyone mutes, or a screaming child nobody trusts, and either way you have lost the thing you built it for.
Where this goes wrong in practice
The most common mistake is not technical. Teams write forty expectations in the first week, covering every column they can think of, then nobody maintains them once the data model shifts three months later. A stale suite that flags harmless changes as failures teaches your team to ignore the tool, which is worse than not having it at all: at least an absent safety net does not lull anyone into a false sense of one. Start with ten checks that map to real incidents, prove they catch something, and expand from there.
The second mistake is treating validation as a project with an end date rather than a habit that has to survive contact with a changing schema. Someone adds a column, a business rule changes, an upstream vendor swaps their date format, and the suite that was accurate in January is quietly wrong by June unless someone owns it. Give the suite an owner the same way you would give a dashboard an owner, and review it whenever the underlying table changes, not on a fixed calendar that has nothing to do with when the data actually moves.
The third is skipping the datasource step we covered above and hardcoding connection details into notebooks instead. It works for a demo. It falls over the moment two people need to run the same checkpoint against two different environments, and you end up debugging a credentials mismatch instead of a data quality problem.
Getting this into the wider architecture
Great Expectations solves validation. It does not solve where your data architecture lets bad data in to begin with, or what happens once a column nobody was watching starts drifting, or how you catch the same class of problem in unstructured data that never had a clean schema to validate against in the first place. We build these systems for clients precisely because the tooling is only ever as good as the pipeline design underneath it: a well-placed checkpoint on a badly designed pipeline still catches the same bug every week, because nobody has fixed the actual cause.
If you already run Great Expectations and it is quietly full of noisy, unmaintained checks, or you are trying to work out where in your pipeline validation should actually sit, talk to us. We would rather help you fix the three checks that actually matter than watch you write another forty that nobody looks at.