A pilot works in the demo. Then someone tries to put it in front of a real customer, and it falls over, because nobody quite trusts the numbers underneath it. That is almost never an algorithm problem. It is a data quality problem, and it is usually the last thing anyone checks.
We see this pattern constantly at Shipshape Data: duplicate customer records, three different date formats sitting in the same warehouse, a "status" field that means five different things depending on which team filled it in. None of that shows up in a slide about your AI strategy. It shows up six weeks into a project, when someone finally queries the underlying tables and the model's confident answer turns out to rest on a foundation nobody actually checked.
Bad data is expensive in ways that rarely get costed properly. Engineers spend hours reconciling numbers that should already agree. Analysts build the same report three times because three teams do not trust each other's version. And once a model gives one visibly wrong answer, people stop trusting the rest of its output too, and that trust is slow to earn back.
This is a practical guide to fixing it: seven strategies we use with clients, in roughly the order that works. Assess first, then govern, then standardise, then validate at the point of entry, then clean what matters, then fix the architecture underneath, then keep watching so it does not slide back. None of these steps requires buying a platform before you understand your problem, which is, frankly, the mistake we see most often.
1. Run an honest assessment before you spend anything
You cannot fix what you have not measured, and most organisations have never actually measured their data quality. They have opinions about it. An assessment replaces those opinions with a picture of where your data lives, how accurate it is, who is responsible for it, and which problems are genuinely blocking the outcomes you care about.
Map what you have before you touch anything
Start by cataloguing every data source feeding your analytics and AI work: the obvious warehouse tables, but also the spreadsheet a finance analyst maintains by hand, the CRM export nobody remembers setting up, the API feed that has quietly been failing for months. Note where each source lives, what format it is in, how often it updates, and who has access. This sounds tedious because it is tedious. It is also the only way to see the whole picture rather than the corner of it your team happens to work in.
Once you have the map, profile the data itself. Look for missing values, duplicate records, outliers that do not make sense, and schema mismatches between systems that are supposed to describe the same thing. Check specific dimensions: is a customer record complete, is it consistent across systems, is it accurate against a source of truth, is it timely enough to be useful. This baseline is what tells you how to improve data quality in your specific environment, rather than applying a generic checklist that misses your actual problems.
Rank the gaps that actually matter
Not every quality issue deserves the same attention, and treating them as if they do is how improvement projects stall. Score each gap on two things: the business impact if it stays broken, and the cost of fixing it. Datasets feeding customer-facing AI, regulatory reporting, or revenue decisions go to the top of the list. A slightly inconsistent internal spreadsheet that three people glance at once a month does not.
We have watched teams spend months polishing a dataset almost nobody uses, while the one feeding their flagship AI feature sat untouched. Prioritising by impact, not by what is easiest to fix, is what gets you visible results in the first few months instead of a tidy but invisible improvement three years from now.
2. Put someone in charge of the data
Data quality fails quietly when nobody owns it. Not dramatically, not all at once. A field starts getting filled in inconsistently, a validation rule gets skipped under deadline pressure, a definition drifts slightly between two teams, and eighteen months later nobody can explain why the numbers do not add up. Governance is what stops that drift, by assigning clear responsibility for accuracy, consistency and fitness for purpose.
Name owners and stewards
Identify specific people, not teams, who own the critical data domains: customer records, product hierarchies, financial transactions, whatever matters most to your business. Owners make the calls on definitions, quality standards and access. Below them, appoint stewards who handle the day to day: monitoring quality, resolving exceptions, being the person a confused analyst can actually message. "The data team owns it" is not ownership. A name is ownership.
Data quality is not a job description. Somebody specific has to be able to say, this is mine, when a number looks wrong.
Decide who decides
Write down who has the final say when two teams define "active customer" differently, or disagree about what a quality threshold should be. Set policies for retention, privacy and compliance, and exception handling. Build an approval path for schema changes and new data sources, so nobody discovers a breaking change after it has already broken three downstream reports. Clear decision rights are what stop governance turning into an endless negotiation everyone quietly avoids.
Make governance part of the job, not a meeting about the job
The fastest way to kill a governance programme is to bolt it onto a monthly meeting people attend without changing what they do the other twenty-nine days. Put quality checks and approval gates inside the pipelines and interfaces people already use: a required field in the entry form, an automated check in the deployment pipeline, an alert that fires before a bad load reaches production. Governance that lives inside the workflow survives. Governance that lives in a slide deck gets forgotten by the second quarter.
3. Agree on what your words actually mean
Inconsistent formats and definitions sabotage analytics before anyone opens a dashboard. If sales calls a customer anyone who has ever filled in a form, and finance calls a customer someone who has paid an invoice, every report that joins the two datasets is quietly wrong from the start. Standardisation gives your organisation one language. Documentation makes sure that language survives when the person who defined it moves on.
Write the definitions down
Define what each critical term actually means: what counts as an active customer, how revenue gets calculated, what makes an order complete. Write it in plain language two departments can both read without an interpreter. When two teams already use the same word differently, do not quietly pick a winner. Either agree a single authoritative definition, or split them into two clearly named attributes. The point is removing the ambiguity that lets two people pull the "same" report and get different numbers.
Line up formats, codes and reference data
Standardise date formats, currency codes, country names and units of measurement across every system that touches them. Build a master list of allowed values for anything categorical: product types, customer segments, status codes. Once teams pull from the same reference list rather than inventing new variants of "Active", "ACTIVE" and "actv", automated checks can actually flag entries that fall outside the standard, instead of drowning in false positives from formatting noise alone.
Keep a catalogue people actually open
Build a searchable record of what data exists, where it lives, who owns it and how to get access. Document lineage too, so people can see which systems feed a given dataset and what happens to it along the way. A catalogue that goes stale the week after launch is worse than no catalogue, because people keep trusting it after it stops being true. Update it whenever a source, definition or integration changes, and treat that update as part of the change rather than an afterthought nobody gets round to.
4. Stop bad data at the door
Prevention beats correction every time. Catching an error while someone is still filling in a form, or the moment a record lands in your pipeline, costs a validation rule and a clear message telling them what to fix. Catching the same error six months later, once it has spread through four downstream systems, costs a data steward the better part of a week. This is the strategy that moves how you improve data quality from firefighting after the fact to catching problems at the source.
Build validation close to where data is created
Put validation directly in the entry forms, APIs and upload interfaces where people and systems first submit information: required fields, sensible value ranges, format patterns, referential checks against records that should already exist. Reject bad entries immediately, with a message that tells the person what went wrong rather than a generic error code. Catching a mistake while the context is still fresh in someone's head is far cheaper than reconstructing that context during an investigation weeks later.
Automate the checks inside your pipelines
Every data pipeline, ETL job and integration should carry its own automated checks: schema compliance, nulls in fields that cannot be null, statistical outliers, logical consistency between related fields, an order cannot ship before it was placed, for instance. When a check fails, stop the pipeline rather than letting questionable data quietly reach production. It feels aggressive the first time a build breaks over what looks like a minor issue. It is considerably less aggressive than explaining to a client why their dashboard has been wrong for three weeks.
Give exceptions somewhere to go
Not everything that fails validation should be rejected outright. Some of it is a legitimate edge case your rules have not caught up with yet. Route failures to a steward who can investigate, correct what needs correcting, and update the rule if the exception turns out to be normal rather than an error. Log every failure with enough context to actually troubleshoot it, and review the pattern of failures periodically. If the same field keeps failing the same check, that stops being a data problem. It is a process problem upstream.
5. Cleanse, deduplicate and enrich the data that matters
Raw data rarely arrives fit for use. Cleansing fixes the errors and inconsistencies that quietly undermine model accuracy. Deduplication removes the records that inflate your counts and confuse anything trying to learn a pattern from them. Enrichment adds the context that was missing in the first place. None of this needs to happen everywhere at once, and trying to make every dataset perfect is a good way to run out of budget before you fix the ones that matter.
Profile before you clean anything
Prioritise the datasets that directly support revenue, compliance or the AI initiatives leadership actually cares about. Profile them for real numbers: what proportion of records are duplicated, how complete are the key fields, how consistent are the formats. Score issues by their effect on the business rather than how technically severe they look on paper. This is what stops cleansing turning into an open-ended project that never quite finishes, because there is always one more field that could be tidier.
Deduplicate, then fix what is obviously wrong
Match duplicate records across several attributes rather than trusting a single ID field, which will always miss the customer who signed up twice with a typo in their email address. When automated confidence in a match is low, put it in front of a person rather than merging blind. A wrong merge is often worse than a duplicate you leave alone. Fix the obviously systematic errors, formatting inconsistencies, invalid codes, values outside a sane range, with automated rules once you understand the pattern. And write down what you changed. A colleague who spots an unexpected transformation six months later should be able to find out why it happened, not have to guess.
Enrichment turns clean data into useful data
Once a dataset is accurate, add the context that makes it valuable: geographic codes, industry classifications, calculated fields like customer tenure or transaction velocity, links between customer records and the transaction history that explains their behaviour. This is the difference between data that is merely correct and data that actually tells you something. Cleansing removes what is wrong. Enrichment adds what was missing.
6. Break down the silos underneath everything
Isolated systems that cannot talk to each other are where most of this starts. Departments build their own tools, those tools cannot exchange data cleanly, and you end up with three versions of the same customer sitting in three databases that disagree with each other. Legacy platforms make it worse by locking data into formats modern AI tooling cannot reach without a translation layer nobody wants to build twice. Fixing quality problems on top of an architecture like that is a permanent tax. Fixing the architecture removes the tax.
Consolidate onto platforms people actually trust
Move critical datasets out of fragmented systems and into a centralised platform, a cloud warehouse or data lake that supports one consistent way of accessing data rather than five slightly different ones. Build integration layers that standardise formats as data moves between systems, instead of leaving every downstream team to handle the inconsistencies themselves. Where it makes sense, put in place a single authoritative source for core entities like customers and products, so "which version is correct" stops being a weekly argument.
Make lineage visible so root causes are findable
Track where data originates, what transformations happen to it, and which systems consume the result. When something breaks downstream, this is what lets you trace it back to the actual cause in minutes instead of days. Visibility here does more than speed up debugging: it lets you assess the blast radius of a proposed change before you make it, rather than finding out afterwards which five reports you just broke.
Design for the workload you actually have
Build platforms that can carry real analytics and model training volumes without slowing the transactional systems underneath them. Give data scientists a way to version and experiment with data safely, away from production. Choose architecture that can grow with your data volumes and with however complex the AI work becomes, because whatever you build now will be under more pressure in two years, not less.
7. Keep watching after the cleanup
Quality improvement that stops the day the project ends does not stay improved. Source systems change, integration logic drifts, new data patterns turn up that your original rules never anticipated. Continuous monitoring protects the investment you already made in the six strategies above, instead of watching it quietly decay over the following year.
Pick metrics that tie back to outcomes
Set measurable indicators for each critical domain: completeness percentage, duplicate rate, how quickly updates land. Then connect those numbers to something a business person actually cares about: model accuracy, report reliability, how confident people feel making a decision off the data. Track both the current level and the trend, because a metric that is stable but mediocre needs a different conversation to one that is dropping fast.
Put it on a dashboard someone actually looks at
Build dashboards that show current quality against defined thresholds, with history so people can see whether things are getting better or worse. Set automated alerts so a steward hears about a quality drop the day it happens, not the day a customer complains. Fold quality monitoring into whatever observability tooling already gets attention in your organisation, so a data problem gets treated with the same urgency as a system outage, rather than sitting in a queue nobody checks.
Close the loop with the people using the data
Ask analysts and data scientists directly what quality problems they run into. Not a survey once a year, an actual conversation. When something goes wrong, look for the root cause instead of patching the symptom for the third time. And when quality does improve, tell the people who depend on that data. Trust, once lost, takes longer to earn back than the original fix took to build.
Where to start
Seven strategies, and no shortcut through them. Assess first, so you are fixing the right things. Assign ownership, so improvements do not quietly reverse the moment nobody is watching. Standardise definitions, so two departments stop arguing about numbers that should already agree. Validate at the point of entry, so bad data never gets in cheaply enough to be worth catching later. Cleanse and enrich what actually matters rather than everything. Fix the architecture underneath, so quality improvements have somewhere stable to live. Then keep watching, because the work does not finish. It just gets quieter.
We built Shipshape Data because we kept running into the same wall with clients: an AI project would stall, and the actual problem sat three layers below the model, in data nobody had ever properly assessed. If any of the seven steps above feel like the gap in your organisation, that is exactly where we start. Talk to us, and we will help you work out which of these seven matters most right now.