The finance department lists 41,000 customers, while the sales department lists 38,000. No one can explain this difference. When the lists are exported and compared, the data shows that the same company appears four times. It is listed once as Ltd and once as Limited. It appears a third time with an invisible trailing space and a fourth time with a spelling error made by a salesperson at a 2019 conference.
Master data management is the practice of establishing a single reliable record for customers, products, and suppliers across all business systems. This process involves maintaining that record even when different departments attempt to modify the data for their own needs. This guide explains golden records, survivorship rules, the four implementation styles, and the governance methods used to prevent duplicate entries.
Master data management is the work of fixing that and then keeping it fixed. Master data is the set of core business entities that show up everywhere: customers, products, suppliers, employees, sites, cost centres. MDM is the practice of agreeing one definition and one authoritative record for each of them, putting a name against that record, and holding the line while a dozen systems quietly pull it apart again.
Shipshape Data builds data foundations for clients. Many AI projects fail because of master data issues rather than modelling problems. Often, stakeholders cannot confirm the exact number of customers a business has. This guide explains the definition of master data and the process of creating a golden record. It describes the four methods of implementing master data management and the difficulties of matching records. It also explains how these factors change when using AI.
Reasons for duplicate customer records
Data fragmentation is not an intentional goal. It happens gradually as a result of many individual, practical decisions.
The sales department purchases a new CRM system because the previous version operated too slowly. The finance department maintains a separate customer list because the CRM lacks the specific data fields required for accounting. When the company acquires another business, it gains a second set of records for existing clients. An operations employee creates a temporary spreadsheet in 2021 that remains a critical part of the workflow four years later.
Each of those decisions was logical at the time. Collectively, they have created a business where the answer to a simple question varies depending on which person is asked.
Every system thinks it's in charge
The main problem is that most systems are designed to be the primary owner of a record. A CRM, a billing platform, a support desk, and an ERP all operate as if they hold the definitive customer data. Each system is correct in its own context because each maintains a version of the record based on its specific requirements. The sales department focuses on the account. The finance department focuses on the legal entity responsible for payment. The support department focuses on the physical location where service is performed. These are three distinct concepts that use the same label. Using the term "customer" for all of them creates confusion.
When versions are not managed, they become inconsistent. An address might be updated in one location but remain unchanged in others. If a company rebrands, some systems may reflect the change while others do not. This issue is similar to how data silos develop, but it is a separate problem. Silos occur when data is inaccessible. Master data issues occur when the same entity is described in different ways across accessible systems.
Someone else is responsible for the cost
The individuals responsible for data fragmentation are seldom the ones who bear the costs. The finance department spends the final week of each month reconciling disparate records. The marketing department incurs unnecessary expenses by sending identical materials to the same company multiple times. A sales representative may enter a meeting without knowing that the client has an active dispute with support, because the two software systems do not share consistent records.
If you ask three different systems for your total customer count, you will receive three different numbers. This discrepancy happens because you have not designated a single authoritative source for that data.
Defining master data
You do not need to apply this process to all information. Treating every data point as master data can cause the entire programme to stop. You should distinguish between three types of data that function in different ways.
Master, transactional, and reference
Master data describes the core entities your business uses. These include customers, products, and suppliers. This data changes infrequently, but is used in a large volume of transactions and is typically stored in multiple systems. The fact that it exists in more than one system is the primary reason to address master data management.
Transactional data records events. An order, a payment, a support ticket, a shipment. There is a lot of it, it's written once and rarely edited, and it points at master data without duplicating it. You do not run MDM on transactions. You run it on the things transactions refer to.
Reference data is the smaller supporting vocabulary: country codes, currencies, units of measure, your own status values. It causes its own particular headaches, usually when two systems use the same code to mean different things, or when somebody adds a status value in one place and the reports downstream silently stop balancing.
A practical test
We use three criteria to determine if an item should be classified as master data. We check if the data is referenced by many transactions, if it changes at a slow rate, and if it is stored in multiple systems. If all three conditions are met, the item is included in the scope. If only one condition is met, we exclude the item regardless of subjective opinions.
Begin with a narrow focus. A programme that starts by addressing customers and products usually succeeds if it is executed correctly. A programme that attempts to manage eleven domains simultaneously often remains in the design phase after a year.
The golden record, and why it's harder than it sounds
The golden record is the single trusted version of an entity. It is created by combining data from every system that contains information about that entity. This process ensures that one customer has one record and that all downstream systems use the same data. The term may suggest a simple administrative task, but it is more complex. Building a golden record requires the business to decide which system is correct when different data sources disagree.
Deciding which value wins
Survivorship rules determine how these conflicts are resolved. You must write these rules for each individual field rather than for the entire record, because different systems are reliable for different types of data. Your billing platform likely contains the most accurate registered address because incorrect addresses cause invoices to fail, leading to quick corrections. Your CRM is the best source for phone numbers because sales staff use them frequently. The ERP system should be the source for credit terms. No single system is the authority for every piece of information. If you treat one system as the sole source of truth, the resulting record will contain incorrect data in several fields.
Certain rules are straightforward. The information that was updated most recently is given precedence. The record that is most complete is given precedence. For a particular field, a designated system always has precedence. All of these rules are logical, yet they occasionally fail. Therefore, the rules need an owner who can decide when the guidelines lack a clear answer.
The record nobody wants to own
This phase frequently leads to programme failures. Survivorship rules constitute a business decision that the technical team cannot decide on alone. When you enquire "whose version of the customer address is the real one", you are prompting two departments to concur that one of them has been keeping inferior data. Such a discussion needs a senior leader capable of making a final decision. If this does not occur, the rules are postponed, the build process halts, and the project turns into a data warehouse that never fulfils its original goals.
Four methods for implementing MDM
This process may be implemented in multiple ways. The optimal approach relies on your control of source systems and the level of disruption the business can tolerate. Most vendors will assert that their own method is the most advanced option. You ought to evaluate those claims carefully.
Registry
The registry style retains each source system in its original location and builds an index. The hub holds identifiers and matching logic to establish that customer 4471 in the CRM and account 90233 in billing represent the same company. It can assemble a golden record on demand without writing data back to the source systems. This option is the least invasive and the quickest to implement. The drawback is that the underlying systems stay unorganised. You have produced an accurate map of the duplication rather than eliminating it.
Consolidation
Consolidation entails copying master data from multiple sources into a central hub. The system cleans and merges the data within this hub for reporting and analytics. The original source systems continue operating unchanged. Organisations frequently select this approach initially because it delivers a reliable customer list without needing staff to alter their existing workflows. Nevertheless, this method does not address operational issues. Salespeople will continue to see duplicate records in their primary systems.
Centralised authoring
In a centralised model, the hub serves as the system where master data is created and edited. The hub then distributes updates to the various systems that consume that data. This approach resolves the underlying problem, yet it demands considerable effort from the organisation. It alters the daily routines of multiple teams and usually requires technical integration for systems not designed to receive external data feeds. When implemented correctly, this model offers a definitive solution to data consistency issues. If the project lacks adequate resources, the business may end up with a hub that many employees disregard.
Coexistence
Coexistence represents the middle approach. Data continues to be authored in the source systems, while changes flow into the hub to be mastered. These updates then flow back to the source systems to correct them. Users keep using their existing tools while the records converge. This is the most common choice in practice. It is also the hardest to manage because bidirectional synchronisation between systems with differing update speeds generates conflicts. These conflicts need manual resolution. If you wish to compare the specific capabilities of platforms in this category, our review of master data governance solutions examines each tool individually.
Matching: the part everyone underestimates
Each style described above relies on the ability to determine whether two records refer to the same entity. This process may seem like a simple mechanical task, yet it is actually complex.
Deterministic matching compares fields for exact or near-exact matches. This includes identical company registration numbers, email addresses, postcodes, and surnames. You should employ this method when you have a reliable identifier because it is fast, easy to explain, and accurate. However, reliable identifiers are less common than expected. Even when a field exists in the data schema, it is often empty in one third of the rows.
Probabilistic matching calculates similarity scores across multiple fields and merges records that exceed a particular threshold. This method handles abbreviations, typographical errors, and extra whitespace. It also accounts for variations in data entry, such as when one system contains "Acme Holdings" and another contains "ACME Hldgs Ltd". This approach requires a manual setting to define the threshold. Choosing this setting involves a direct trade-off between different types of matching errors.
False merges are worse than false splits
If you raise the threshold, you will create false splits. This causes the same customer to exist as two separate records, reproducing the original problem after the project is complete. If you lower the threshold, you will create false merges where two different companies are combined into a single record. Both outcomes constitute errors. They do not carry the same level of risk.
A false split is embarrassing and easy to correct. A false merge blends two companies' contacts, orders, and credit history into a single record, and unpicking it afterwards is genuinely difficult, because you have thrown away the information that told you which row came from where. Anyone who has tried to reverse one at scale learns to hold the threshold conservative and route the uncertain cases to a person. Which means budgeting for that person. A stewardship queue with nobody working it is just a list of problems with a nicer interface.
Matching quality depends on the inputs, so an engine directed at fields that were never validated on entry will underperform regardless of how good the algorithm is. That makes basic data quality work a prerequisite rather than a parallel workstream.
Governance is what stops it decaying
MDM projects often fail in a predictable manner. The data cleanup is completed successfully, golden records are created, and stakeholders are satisfied. However, after roughly eighteen months, duplicate records reappear. There were no technical errors. The reason is that the project concluded without a designated owner to maintain it.
The solution is practical and focuses on personnel. One person is responsible for each master data domain and has the authority to resolve disputes over definitions. Data stewards manage daily tasks such as processing the match queue, approving new records, and rejecting those that fail to meet standards. Definitions are documented in an accessible location. If a definition for a customer exists only in one person's memory, that information is lost when they leave the company.
Rules are enforced at the point of data creation rather than during later audits. This includes making required fields mandatory and using dropdown menus to prevent spelling variations in free text. Duplicate checks occur when a salesperson creates an account instead of appearing in a monthly report. This approach follows the discipline of data governance. MDM is the stage where governance moves beyond documentation and provides visible value. Using data lineage to trace the source of a mastered value explains why the hub might differ from the data in a specific system.
What changes when AI sits on top
These factors were significant before models were applied. They are more important now, and they frequently cause unexpected problems for teams.
If you ask an AI assistant for your sales volume to Acme, the system must be able to identify the correct company. If your database contains four separate records for Acme, there are four possible answers. The assistant will typically provide a single number without explaining that it only represents a portion of the total account. A human reviewing a report would likely notice a low figure and investigate the discrepancy. A language model will not do this. It provides an answer based on the specific records it retrieves. The confidence of its response does not indicate whether the underlying entity resolution was accurate.
Master data issues often appear early in AI projects, frequently as an unexpected challenge. A retrieval system gathering information about a supplier must collect all relevant data rather than a partial set stored under multiple IDs. When a model is trained on entity-level features, it treats duplicate records as distinct data points. A single customer represented by three records appears as three smaller customers. This situation distorts segmentation and churn prediction in a manner that is hard to identify by looking at the results.
You don't have to finish a multi-year master data management programme before using AI. You need to identify the data entities required by your AI use case and check whether that data is accurate and managed at present.
Where to start
Pick a single domain. Opt for the one that creates clear problems rather than the one that appears easiest to model. Typically this is the customer domain, or occasionally the product domain. If a particular reconciliation task consumes a full week of an employee each month, begin with that domain.
Examine your data prior to starting the design process. Find out the total record count, the degree of overlap, and the level of duplication. The real quantity of duplicate data typically exceeds initial estimates. Having these precise numbers lets you ground governance discussions in facts instead of opinions. Once the data has been analysed, set clear definitions and work with business stakeholders to draft data survivorship rules. Choose the simplest implementation approach that solves the specific problem. Determine who is responsible for the data before you begin building the system.
Master data is the least exciting element of a data foundation and also one of the most load-bearing. If you are evaluating an MDM programme, or you have AI plans that rely on a single view of the customer that you are not certain you possess, talk to us. We would prefer to help you scope it honestly now rather than see a model confidently report on only a quarter of a customer.
If this issue is currently on your roadmap, our data consultancy offers services for this particular task. We start by showing the capabilities of your existing data.
Frequently asked questions
What is master data management?
Master data management (MDM) involves creating a single reliable record for key business entities like customers, products, and suppliers. The aim is to keep this record intact even when different systems try to change the data. This guide notes that most MDM programmes fail because stakeholders cannot agree on which record version is accurate.
What are the differences between master data management and standard data management?
Data management typically includes how you store, secure, and transfer data. Master data management is more focused; it applies specifically to a limited set of entities such as customers, products, and suppliers that appear across many systems and need a single agreed definition. It does not cover every dataset a business maintains.
What are the differences between master data management and database management?
Database management involves operating and querying data within relational systems. Master data management creates a single authoritative version of a business entity, like a customer or product, across all systems that store that entity. This guide notes that this consistency is kept through survivorship rules and stewardship rather than solely by database design.
Is a master's degree in data science required to manage a master data management programme?
No. This guide views MDM as a discipline built on clear ownership, defined survivorship rules, and matching logic. The hardest part is deciding which version of a customer record should be primary. This is a business decision that needs a senior owner rather than a data science expert.
What does master data management, or MDM, involve in a professional context?
In practice MDM involves selecting one domain, typically customer, profiling the true amount of duplication, and then choosing the lightest implementation style, such as registry, consolidation, centralised authoring or coexistence, that solves the actual problem. This guide makes clear that starting narrow tends to succeed, whereas trying to master several domains at once tends to stall in design workshops.