Data quality: what it is, why it matters and how to improve it

A report goes out with the wrong numbers. A customer gets billed twice for an order they placed once. A model trained on last year's data starts recommending a product line you stopped stocking in March. Three different failures, one shared cause: the data behind them was never good enough to trust.

Data quality is how well your data holds up against a set of standards, accuracy, completeness, consistency, timeliness, validity, and uniqueness, rather than a single yes-or-no property. Good data quality means a customer's address is correct and a record exists once rather than three times under three spellings. Poor data quality makes every one of those things uncertain, and uncertainty is expensive.

Data quality is a set of standards rather than a single yes-or-no property: accuracy, completeness, consistency, timeliness, validity, and uniqueness. How well your data holds up against each one decides whether you can act on it with confidence. Good data quality means a customer's address is correct, a sales figure ties back to the transactions behind it, and a record exists exactly once rather than three times under three near-identical spellings. Poor data quality means all of that is uncertain, and uncertainty is expensive.

This guide covers what data quality means in practice, why it costs more than most organisations admit, the dimensions that define it, and how to build the standards, checks and ownership that stop it decaying again the moment you look away. We build data foundations for clients at Shipshape Data, and with almost every one of them, the first conversation is about whether the data itself can be trusted yet, rather than which tool to buy.

Why bad data costs more than you think

Data problems rarely show up as a single dramatic failure. They show up as a slow tax on everything your business does, paid in small amounts by almost every department, which is exactly why the total is so easy to underestimate.

The money you can see

Duplicate records in your CRM cost you twice for the same lead. Marketing spend gets wasted chasing contacts who moved house two years ago. Shipments go out to addresses that were never updated after a customer relocated, and someone has to pay for the return. Gartner has put the average annual cost of poor data quality at £9.7 million per organisation, and whatever your own number turns out to be, it will include the obvious costs, wasted spend, failed deliveries, and the far larger hidden ones: hours your team spends manually correcting errors, chasing down which figure is right, and rebuilding trust after a mistake that a basic validation rule would have caught.

The friction you can't see on a spreadsheet

Bad data doesn't stay contained in one system. It ripples. Sales loses a deal because the contact record had the wrong job title on it. Finance spends the last week of every month reconciling two reports that should agree and never quite do. Customer service takes the call from someone who received the wrong order, again, because the address field and the shipping field disagree with each other. Every one of those teams eventually builds its own workaround, a shadow spreadsheet, a manual double-check, a "just call them to confirm" step, and those workarounds quietly fragment the data further. We have sat in enough of these conversations to know the pattern: nobody set out to build five versions of the customer list, it just accumulated one workaround at a time.

Decisions built on numbers nobody checked

The most expensive failures happen higher up. A board approves an acquisition based on a customer count that turns out to be inflated by duplicates. A product gets more investment because a reporting error makes it look more profitable than it is. A market entry decision leans on demographic data that was accurate five years ago and stale today. None of these are data problems in the narrow sense. They are strategy problems wearing a data-quality costume, and they get harder to spot the more senior the decision-maker is, because nobody above a certain level tends to ask where the number came from.

Poor data quality turns your best people into detectives instead of decision-makers, and every hour spent verifying a number is an hour not spent acting on it.

The gap widens further once AI and automation enter the picture, because both amplify whatever quality already exists in the data feeding them. A human analyst might notice that a figure looks wrong and quietly double-check it. A model doesn't do that. It learns the pattern, including the errors, and repeats them at scale.

The dimensions that decide whether your data is good

"Data quality" sounds like a single score, but treating it that way is where most improvement programmes go wrong. It breaks down into distinct dimensions, and your organisation needs to weigh them differently depending on how the data gets used. A marketing database and a set of financial records don't need the same standard of accuracy, and pretending otherwise wastes effort on the wrong problem.

Accuracy and validity

Accuracy is whether the data reflects reality. A postcode that matches where the customer lives, an order total that equals the sum of its line items, a product spec that describes what you actually ship. Get this wrong and you send goods to the wrong place, bill someone incorrectly, or make a call based on something that simply isn't true.

Validity is narrower: does the data follow the format your systems expect? An email address needs an @ symbol and a domain. A date needs to follow one format consistently rather than switching between DD/MM/YYYY and MM/DD/YYYY depending on who typed it in. Invalid data doesn't necessarily lie to you the way inaccurate data does, but it breaks automated processes just as effectively, because a system expecting a clean date field will choke on one that occasionally isn't.

Completeness and consistency

Completeness is simple to define and hard to enforce: do the required fields have values in them? A missing phone number stops your sales team following up. A blank delivery instruction field causes a shipment delay nobody saw coming. An empty product description tanks a conversion rate that marketing then gets blamed for. The fix isn't complicated in principle, decide which fields are mandatory for which process and enforce it at entry, but it takes discipline to apply.

Consistency is the same information agreeing with itself across systems. Your customer's name should read the same way in your CRM, your billing platform, and your support desk. The moment two systems disagree about something as basic as a name, someone has to decide which one is right, and that decision eats time every single time it comes up.

Timeliness and uniqueness

Timeliness is about whether data is current enough for what you are using it for, and that threshold moves depending on the use case. Real-time inventory needs to be accurate to the minute. Annual planning can tolerate demographic data that is a few months old. The mistake is applying one freshness standard everywhere, which either wastes effort refreshing data nobody needs refreshed that often, or leaves genuinely time-sensitive data stale.

Uniqueness means each record exists once. Duplicate customer profiles inflate your metrics, get the same person three marketing emails in a week, and waste budget you could have spent on someone who doesn't already have three profiles in your system. Left alone, duplicates multiply. A system with no deduplication logic in place tends to get worse, not better, over time.

Building data quality into how you work

None of this gets fixed with a one-off clean-up. A spreadsheet gets scrubbed, everyone feels better for a month, and the same problems creep back because nothing changed about how data enters the system in the first place. Lasting improvement needs three things working together: standards that define what good looks like, automated checks that catch errors before they spread, and someone whose job it is to care.

Standards you enforce, not just write down

Start by deciding, explicitly, what counts as acceptable data for each field and system: how a customer name gets formatted, which address fields are mandatory, what a valid email address looks like, how dates get recorded. Write it down somewhere your teams will find it when they build a form or design a database, because unwritten standards mean every department invents its own version, and that is exactly how you end up with five different date formats across four systems.

Writing the standard down is the easy half. The rules only do anything once they are enforced at the point of entry: reject data that fails validation, force a dropdown where free text invites typing errors, make fields mandatory before a record can save. A rule enforced automatically stops bad data getting in. A rule written in a document and never checked stops nothing, and six months later someone finds out the hard way.

Automated validation and monitoring

Manual checking doesn't scale. Nobody is reviewing every row by hand once you are past a few thousand records, and real-time pipelines move faster than any manual process could keep up with anyway. You need automated tools scanning continuously for duplicates, missing values, outliers, and inconsistencies, and routing the alert to whoever is responsible for that dataset rather than into a shared inbox nobody checks. Automation doesn't replace judgement. It replaces the tedious first pass that judgement should never have had to do in the first place.

Ownership through governance, not committees for their own sake

Data quality improves when a specific person owns it, and stalls when everyone assumes someone else does. Assign data stewards for the datasets that matter, people with the authority to reject poor data and the standing to escalate when a problem keeps recurring rather than getting patched over again. A governance structure above that, setting organisation-wide standards and resolving disputes between departments, matters less for the paperwork and more for the fact that it makes ownership unambiguous. The goal is to make sure a data quality problem has exactly one person's name attached to it, not zero and not five, rather than to create another committee for its own sake.

Measuring data quality properly

You can't improve what you haven't measured, and "the data feels a bit off" isn't a metric anyone can act on. Turning data quality into numbers lets you see which datasets cause the most damage, justify spending time fixing them, and show whoever controls the budget that the investment worked.

Pick a metric for each dimension that matters

Accuracy can be measured as the percentage of records that match a verified source of truth. Completeness is the share of fields that have a value in them rather than sitting blank. Consistency is how often the same piece of data agrees with itself across the systems that hold it. None of these need to be complicated to calculate, and a rough baseline beats no baseline every time.

Set a target for each one, and let the target vary by dataset. A marketing database might be fine at 95% accuracy. Financial records need something closer to 99.9%, because the cost of being wrong isn't the same. Write the targets down so your team knows what "good enough" actually means for the specific dataset they are looking at, rather than guessing.

Roll it into one score, carefully

A single overall quality score, weighted by how much each dimension matters for that dataset, gives you something you can compare across systems and track over time. Accuracy might carry 30% of the weight for a customer database and timeliness only 10%; a live pricing feed would weight that the other way round. The number itself matters less than what it lets you do: spot which dataset is dragging the average down and put effort where it helps.

Audit on a schedule, not when something breaks

Automated monitoring catches the obvious problems. A scheduled audit, monthly or quarterly for your most critical datasets, catches the ones that slip through: sample records at random, check them against a source document or an external reference, and log what your automated checks missed. This is usually where you find the actual root cause, a form with no validation on it, an integration between two systems quietly dropping a field, rather than just another list of records to fix one at a time.

What this means once AI enters the picture

Everything above matters more, not less, once machine learning or automation sits on top of your data, because AI amplifies whatever quality already exists rather than correcting for its absence. A human reviewing a report might notice a number looks strange and go check it. A model trained on that same number learns it as fact and repeats the pattern at whatever scale you deploy it.

What training data needs

A model needs enough clean, representative data to learn patterns that hold up outside its training set, not just inside it. That means covering the edge cases and rare scenarios the system will meet in production, verifying labels are correct before anyone trusts a supervised model built on them, and removing outliers that would skew what "normal" looks like to the model. Skip any of that and you get a biased system: one that under-serves certain customer segments, mishandles inputs it never saw enough examples of during training, or produces answers confident enough to sound right and wrong enough to cause a problem before anyone notices.

Quality doesn't stop once the model is live

Production systems keep processing new data long after training finishes, and that data needs the same discipline. Validation checks should reject inputs that fall outside expected ranges. Something should be watching for drift, the signal that live data has quietly diverged from what the model was trained on. And someone needs to look when confidence scores start dropping, rather than waiting for a customer to notice first. Model performance tends to degrade quietly. Accuracy slips a percentage point at a time, and by the time it is obvious in the business numbers, it has usually been wrong for a while.

Where to start

Start with the dataset that causes the most pain right now, not the one that is easiest to fix. Audit it properly, put automated validation and monitoring around it, and give it an owner with the authority to keep it clean once the initial clean-up is done. That single dataset, done properly, will teach you more about your own data than a company-wide policy document ever will, and the pattern you build for it is the one you repeat everywhere else.

Clean, well-governed data is the foundation everything else gets built on, especially anything involving AI. If you are trying to work out where your own data stands, or you want a second opinion before committing budget to a data quality programme, talk to us. We'd rather tell you honestly where the gaps are now than watch you find them during an audit in a year's time.

Frequently asked questions