Data migration framework: a step-by-step plan that holds

A migration plan usually starts with a target date and a list of tables. It rarely starts with the business rules buried in old stored procedures, the duplicate records everyone works around, or a shared view of what finished looks like. Those gaps decide whether people trust the new system on its first working day.

A data migration framework is a structured, repeatable plan for moving data from a source system to a target system without losing meaning, accuracy or trust. It sets out the stages, from assessment through mapping, build, reconciliation, cutover and support, and names who signs off each one and what evidence counts as done.

This guide is for the people who own a migration: heads of data, finance and operations leads, and technology directors who have been given a date and a new platform. It explains why migrations go wrong, sets out a six-stage framework, and covers the choice between a big bang and a phased approach. It also explains where data migration tools help and how to run the work across several teams. We build data foundations and migrate data onto them, so the advice comes from delivery work rather than theory.

What data migration involves

Data migration is the permanent transfer of data from one storage system, format, database or application to another. Common reasons include replacing an ERP or CRM, retiring an on-premises database, consolidating systems after an acquisition, or moving reporting onto a cloud data warehouse. If the data move is part of a wider infrastructure change, a cloud migration roadmap covers how to sequence the applications and workloads around it.

A migration is different from a one-off export. On a set date the old system stops being the record of truth, and the new one must carry the same meaning from that point. The work is therefore about definitions, rules and evidence as much as about moving rows.

The four common types

• Storage migration: moving data between storage hardware or services, with the format largely unchanged.
• Database migration: moving data from one database engine or version to another, often with schema changes.
• Application migration: moving data into a new business application, such as a new ERP, CRM or finance system.
• Cloud migration: moving data from on-premises systems into a cloud platform or warehouse.

Most real programmes combine two or more of these. A move from an on-premises ERP to a cloud ERP is both an application migration and a cloud migration, and it needs the discipline of both.

Why migrations go wrong

Copying data from one place to another is rarely the part that fails. The failures sit in work nobody planned for. Three causes account for most of the delays and disputes we see.

Business rules nobody wrote down

Older systems carry logic that exists nowhere else. A stored procedure sets a discount flag. A field means one thing for UK accounts and another for EU accounts. A status code was retired years ago, but a team still filters it out in a spreadsheet. If nobody records these rules before mapping starts, the new system loads the values and loses their meaning. Reports then disagree with the old ones, and nobody can say why.

Quality problems carried across

Duplicate customers, missing postcodes, free-text dates and orphaned records do not fix themselves in transit. A like-for-like copy moves every one of them into a platform that people expect to be cleaner. Profiling the source against the standard data quality dimensions early tells you which problems to fix before the move, which to fix during it, and which to accept and record.

No agreed definition of done

Engineering teams often treat a migration as finished when the load job completes. Finance treats it as finished when the ledger balances. Operations treats it as finished when the morning reports match. Without one written definition, signed by the people who own the data, cutover turns into an argument at the worst possible moment.

A migration is finished when the business can prove the numbers agree, not when the load job stops running.

The framework, step by step

The framework has six stages. Each stage ends with evidence that the next one can start. The stages overlap in practice, because mapping uncovers new rules and testing uncovers new quality problems, but the sign-offs should stay separate.

1. Assess the source and the target

Start with an inventory: every source system, its volumes, its owners, and everything that consumes its data, including reports, integrations and models. Profile the data to find gaps, duplicates and inconsistent formats. Record every business rule you find in a rule register. Then decide scope: what moves, what is archived, and what is left behind. The outputs are a scope document, a quality baseline, the rule register and a first risk log.

2. Map fields, rules and definitions

Mapping sets out, field by field, where each source value goes and how it changes on the way. It covers keys, reference data, code lists and transformation rules. This is also where business definitions get agreed, so that "active customer" means the same thing in the new system as it did in the old one, or a documented new thing. A data dictionary is the natural home for those definitions. Data owners, not only engineers, should review and sign the mapping.

3. Build and test the pipelines

Build the extract, transform and load steps as version-controlled, repeatable code rather than one-off scripts. Run the full migration many times against copies of production data. Each trial run tests volumes, timings and failure handling, and each one should be faster and cleaner than the last. Record where every value came from and what happened to it. Data lineage turns a disputed figure after go-live into a question you can answer in minutes.

4. Reconcile, row by row

Reconciliation proves the target agrees with the source. It works at several levels: record counts per table, totals by key dimension such as region or account, field-level comparisons, and business checks such as balances that must tie out. Agree tolerances before testing starts, not after. Every difference must be explained, fixed or formally accepted by its owner. Our guide to data reconciliation covers the methods in more detail.

5. Cut over with a rollback plan

Rehearse the cutover end to end at least once, with the real timings. Agree the freeze window, the order of steps, the final reconciliation, and the go or no-go criteria in writing. Prepare a rollback plan and decide in advance who can trigger it. Tell users what will change, when, and who to contact if something looks wrong.

6. Support the new system after go-live

The first weeks after go-live, often called hypercare, are when hidden rules surface. Keep the migration team in place, monitor loads and reports, and triage issues daily. Continue reconciliation through at least one full business cycle, such as a month-end close. Retire the source system only when the new one has held up through that cycle, and keep a read-only archive if retention rules require it.

Big bang or phased: choosing a strategy

A data migration strategy sets how the move happens over time. The two main options are a big bang, where everything moves in one cutover, and a phased approach, where data moves in stages.

Big bang migration

In a big bang migration, the old system stops, all data moves in one window, and the new system goes live. It avoids running two systems side by side and keeping them in step. It also concentrates the risk: if reconciliation fails during the window, the whole cutover may need to roll back. It suits smaller data volumes, systems with few integrations, and organisations that can accept a planned outage.

Phased migration

A phased migration moves data in stages, by module, business unit, region or data domain. Each phase is smaller and easier to reconcile, and the team learns from each one. The cost is complexity: the old and new systems must coexist, which usually means temporary interfaces or synchronisation. Some programmes add a period of parallel running, where both systems operate and their outputs are compared before the old one is switched off.

How to decide

• Tolerance for downtime: if the business cannot stop for the length of the load, a phased approach is more realistic.
• Data volume and complexity: larger, more interconnected estates usually benefit from smaller, reconcilable phases.
• Number of integrations: each connected system adds coexistence work in a phased plan and cutover risk in a big bang.
• Team capacity: phased plans need people for longer; a big bang needs more people at once.
• Regulatory and reporting deadlines: avoid cutovers that land on a year-end, a filing date or a peak trading period.

The choice also shapes the budget. Our guide to what drives data migration cost explains how strategy, volume and data quality affect effort.

Where migration tools and software fit

Data migration tools and software cover a wide range: connectors that extract from source systems, replication and change data capture services, transformation frameworks, validation and testing tools, and schedulers. Cloud providers offer their own migration services, and many business applications ship with import frameworks of their own. Used well, these tools make each trial run faster and more repeatable.

What tools do well

• Extract data reliably from common databases and applications.
• Move large volumes and capture changes made during the migration window.
• Run transformations as tested, repeatable code.
• Automate counts, checksums and rule-based validation.
• Schedule, log and rerun jobs.

What tools cannot do for you

• Find business rules that only exist in people's heads or old spreadsheets.
• Decide what a field means, or which of two conflicting records is correct.
• Agree the definition of done with finance, operations and data owners.
• Choose what to archive and what to leave behind.

Choose tools after assessment and mapping, not before. The source systems, the target platform, the volumes and the reconciliation method should drive the choice. Buying a tool first often means bending the plan around it.

Running a migration programme across teams

A data migration programme that touches several departments needs clear ownership. Engineers can move data, but only the business can say whether the result is correct. Set up the roles before the first mapping workshop.

• Sponsor: one senior person who owns the outcome, settles priority conflicts and approves go or no-go.
• Data owners: one per domain, such as customers, products or finance, who sign off mappings and reconciliation results.
• Migration lead: runs the plan, the risk log and the decision log day to day.
• Engineers: build and run the pipelines and the trial migrations.
• Testers and business users: run user acceptance tests against real scenarios.

Keep one plan, one risk log and one decision log that every team can see. Hold a short weekly review where open rules, mapping questions and reconciliation differences are decided rather than discussed again. Record every decision with its owner and date, so that nobody has to reopen it during cutover.

Plan for the fact that data owners also have day jobs. Mapping reviews and reconciliation sign-offs are where programmes slow down, so book that time early and protect it.

Use the move to land on a governed foundation

A migration forces you to profile every source, document every rule, agree every definition and test every value. That is most of the work of building a governed data foundation. Doing it once, during the migration, costs far less than copying today's problems across and fixing them later in a system people already rely on.

A migration is the one point where fixing the foundation costs less than copying its problems across.

In practice, that means landing the data in stages. A bronze layer holds a raw copy of each source, so you can always trace back. A silver layer holds cleaned and conformed data, with duplicates resolved and formats standardised. A gold layer holds business-ready tables built on agreed definitions. A semantic layer on top gives everyone one governed definition of each business term, so reports and models draw on the same meaning. Core records such as customers and suppliers often need a master data management approach at this stage.

This matters more if AI is on the roadmap. AI amplifies whatever it sits on. A model built over migrated data that still carries duplicates and undocumented rules will produce confident, wrong answers faster than any report did. A migration that lands on a governed foundation gives later AI work something it can trust.

Where this leaves you

A data migration framework turns a date and a list of tables into a plan with evidence at every stage. Assess the source honestly, map rules and definitions with the people who own them, build repeatable pipelines, reconcile until every difference is explained, rehearse the cutover, and stay close after go-live. Choose big bang or phased based on downtime, complexity and capacity, and choose tools once you know what they need to do.

The bigger decision is what the migration is for. You can copy the old system's problems into a new platform, or you can use the work you are already doing to land on a foundation your teams and future AI work can rely on. The second option costs less now than it ever will again.

Our data migration service moves warehouses, databases and cloud workloads onto a governed foundation, reconciled row by row and supported past cutover. Where the target is a new platform, we build the data foundation first, so the migrated data lands cleaned, defined and traceable. We scope honestly, name what the result depends on, and say if we are not the right fit. Talk to us to plan the migration around your systems before the date is fixed.

Frequently asked questions