A fraud model goes live in January catching nine cases in ten, and the team that built it moves on to the next thing, because that's what a good launch looks like. By September the same model is catching six in ten. Nobody touched the code. Nobody redeployed it. The weights are exactly what they were the day it shipped. What changed is the world the model is now looking at, and the model has no way of noticing that on its own.
Model drift is what happens when a machine learning model's accuracy quietly erodes after launch, even though its code hasn't changed. The world it scores keeps moving, customer behaviour shifts, and a frozen model has no way to notice. Catching it needs monitoring on inputs and outcomes, plus ruling out a broken pipeline first.
We build data and AI systems at Shipshape Data, and this is one of the quieter ways an AI project goes wrong. It rarely happens at launch, when everyone is watching. It happens months later, when the dashboard still says the model is running and nobody is checking whether it's still right. Teams plan for the build. Fewer plan for the slow, unannounced decay that starts the moment the model stops learning and the world keeps moving.
This guide covers what model drift is, the difference between data drift and concept drift, how to catch either one before customers do, what a monitoring and alerting setup should look like day to day, how to think about retraining cadence, and the question we get asked most often on this topic: how do you tell a drifting model apart from a pipeline that has quietly broken.
Why a model that worked at launch stops working
A model is a frozen snapshot of a pattern that existed in the data it was trained on. Nothing about it updates once it's deployed, unless you explicitly retrain it. The world it's scoring, on the other hand, keeps moving: customers change habits, suppliers change prices, fraudsters change tactics, a pandemic happens, an economy tips into a recession. Model drift is the gap this opens up between the frozen pattern and the moving target, and it shows up as accuracy that erodes without any change to the model itself.
Some of this is genuinely unavoidable. A model trained on three years of stable trading conditions has no way to anticipate a regulatory change that lands next quarter, and no amount of careful engineering at build time changes that. Drift itself isn't the mistake. The mistake is treating a launch as the finish line, when a launch is closer to the starting gun for an ongoing job that nobody assigned to anyone.
It helps to split the gap into two different things, because they are caused by different problems and fixed in different ways.
Data drift: the inputs change shape
Data drift is a change in the inputs the model sees, while the underlying relationship between those inputs and the outcome stays the same. A retailer's average basket size creeps up because prices rose, not because shopping habits changed. A demographic shift moves the average age of loan applicants. An app redesign changes which button people click first, which changes the shape of a clickstream feature that has nothing to do with intent. The model still knows the right relationship. It's just being handed inputs that look increasingly unlike what it trained on.
Concept drift: the relationship itself moves
Concept drift is nastier, because the inputs can look perfectly normal while the relationship between them and the outcome has changed underneath. The clearest example is fraud. A fraud detection model learns the patterns fraudsters used last year. Fraudsters don't stand still, because there's money on the other side of the model, and once a pattern is known to be caught, it stops being used. The features feeding the model can look exactly as they did at launch. What used to predict fraud no longer does, and something that used to look innocent now isn't.
Both kinds are normal. A model that never drifted would mean the world had stopped changing, which isn't the world anyone operates in.
A recommendation engine shows both at once, which is part of why retailers get so used to retraining. Data drift arrives every January when the sale ends and browsing behaviour resets. Concept drift arrives more quietly, when a competitor undercuts a category and price stops being the thing that predicts a purchase the way it did in November. Same model, same dashboard, two different diseases.
Telling the two apart in practice
The distinction matters because the fix is different. Data drift is often survivable with the existing model, at least for a while, because the relationship it learned is still true. Concept drift means the model is actively learning the wrong thing every time it scores a new case, and no amount of waiting fixes that. It needs new labels and a retrain, sooner rather than later.
In practice you rarely get a clean signal telling you which one you are looking at. A demand forecasting model that starts missing during a supply shock could be either: the inputs shifted (data drift, prices and lead times moved) or the relationship shifted (concept drift, customers now behave differently under scarcity than they did when supply was normal). Most real drift events are a mix. The useful habit is checking both the input distributions and the model's error pattern whenever performance slips, rather than trying to label every incident perfectly, because a distribution check that comes back clean while errors climb is itself informative: it points you towards concept drift and away from data drift, which changes what you do next.
How to detect drift before your customers do
The honest starting point is that most organisations find out about drift from a complaint, a missed forecast, or a quarterly review, rather than from a monitoring system. That's backwards, and it's also completely normal, because building drift detection well takes more than most teams expect on day one.
Watching the inputs
Population stability index and the Kolmogorov-Smirnov test are the two workhorses for comparing a current batch of inputs against the training distribution, feature by feature. Neither is exotic. PSI gives you a single number per feature that flags when a distribution has shifted meaningfully; the KS test tells you whether two samples plausibly came from the same distribution at all. Run either on a schedule, plot the trend, and you catch data drift while it's still small, rather than after it has compounded for six months.
Watching the outcomes
Input checks miss concept drift, because the inputs can look fine while the relationship has moved. For that you need to track the model's actual performance over time: accuracy, precision, recall, whatever the model was built to optimise. The complication is that ground truth often lags. A loan default model doesn't know if a loan was a bad decision for months. A churn model doesn't know if someone churned until they have gone. When labels are slow, watch proxy signals instead: prediction confidence drifting lower on average, the ratio of positive to negative predictions shifting when nothing in the business explains why, or a rising share of predictions sitting right on the decision boundary, which is often the first visible sign that the model is less sure of itself than it used to be.
None of this requires custom tooling to start. A cron job that computes PSI on a handful of features and writes the result to a spreadsheet is a genuine monitoring system, and it beats an ambitious platform rollout that stalls in procurement for two quarters. If the workload grows past what a spreadsheet can carry, that's when it's worth comparing dedicated options, and our guide to AWS SageMaker model monitoring covers what setting up drift alerts on a managed platform involves, rather than just what the sales page claims.
Monitoring and alerting that people actually trust
Detection only works if somebody sees it and believes it. This is where a lot of drift monitoring quietly fails: the metric exists, the dashboard exists, and nobody looks at it, because it fired three false alarms in its first month and got muted.
Thresholds that don't cry wolf
A PSI threshold copied from a blog post, applied without checking what normal variation looks like for your own data, will either alert constantly on noise or stay silent through a real shift. Set thresholds from your own history: look at how much the metric moves between two ordinary weeks, and set the alert meaningfully above that, not at a round number that felt sensible. Then let a threshold breach earn attention rather than automatic action. The first alert on a new monitor should go to a person, not trigger a retrain, because you are still learning what your own noise floor looks like.
Alerting on the right thing
Alert on the business metric where you can, not only the statistical one. A recall drop from 92% to 84% means something concrete: a specific number of missed fraud cases, a specific revenue exposure. A PSI score of 0.31 means nothing to the person reading the alert unless they already know what PSI is. Translate the statistical signal into the number the business cares about, and route it to whoever owns that number rather than only the team that built the model. Dedicated MLOps platforms do this reasonably well out of the box. They are still only as good as the thresholds and the routing someone configures on top, and a well-configured alert on a spreadsheet beats a badly configured one on an expensive platform.
Retraining strategy and cadence
There are three broad ways to decide when to retrain, and most mature setups end up running some blend of the first two.
Fixed schedule
Retrain weekly, monthly, quarterly, whatever cadence matches how fast your domain moves. It's simple to run and simple to explain to a stakeholder. Its weakness is obvious once you say it out loud: a fixed schedule retrains on a calm month exactly as often as a month where the ground shifted under the model, which means it's sometimes wasteful and sometimes too slow.
Triggered retraining
Retrain when a monitoring threshold fires, rather than on a clock. This responds to actual drift rather than the passage of time, and it's the more efficient approach once your monitoring is trustworthy enough to act on. The catch is right there in that sentence: it only works if the monitoring is trustworthy, which is the entire previous section. Triggered retraining bolted onto noisy alerts just means retraining on noise.
Shadow testing before you promote anything
Whichever cadence triggers a retrain, don't push the new model straight to production. Run it in shadow, scoring live traffic silently alongside the current model, and compare the two on real outcomes before the new one takes over. A retrained model isn't automatically better. It has just seen more recent data, and more recent data isn't always cleaner data. We've seen a retrain go out that quietly encoded three weeks of a broken upstream feed as a new pattern worth learning, because nobody shadow tested it against the model it was replacing.
A model in production is a bet that the world will stay roughly the shape it was on training day, and every month that bet goes unchecked is a month you are trusting it on faith alone.
For most teams the practical answer is a fixed floor (retrain at least quarterly, so nothing goes a year untouched) with triggered retraining layered on top once the monitoring earns that trust. Neither replaces the other.
Is it drift, or is it a broken pipeline
This is the question we get asked most, usually by someone who has just watched accuracy fall off a cliff overnight and is trying to work out whether to call a data engineer or a data scientist. The honest answer is that a sudden, sharp drop is rarely drift. Drift is gradual almost by definition, because the world changes gradually. A model that was fine on Tuesday and badly wrong on Wednesday almost always has a pipeline problem behind it, not a philosophical one.
The tell is the shape of the change
Genuine drift moves like a slope: performance degrades over weeks or months, tracking a real shift in the world you could plausibly explain if you went and looked. A pipeline bug moves like a cliff: a metric that was stable for months falls off in a single day, or a feature's distribution jumps in a way no real-world change explains. If a feature that used to range from 0 to 100 suddenly shows nothing but zeros, that's a null-handling bug in an upstream ETL job, not customers suddenly behaving differently. If a categorical field that had eight sensible values overnight has one new value showing up in forty percent of rows, something upstream changed its schema and nobody told the model.
Check the pipeline first, always
Before touching the model, check whether the inputs it's receiving even mean what they used to mean. Has a source system changed a field's format. Has a currency conversion started applying twice. Has a join upstream started silently dropping rows because a key format changed. A stable, well-documented data pipeline with real data lineage makes this check quick: you can see exactly where a feature came from and what changed along the way. A pipeline nobody has mapped turns the same check into a day of grepping through logs, which is usually the actual reason teams reach for "the model must be drifting" without checking first, because it's genuinely easier to say than to trace six upstream systems.
The two problems can also compound. A pipeline bug can look enough like data drift that a team retrains around it, baking a broken feed permanently into the new model instead of fixing the feed. That's a worse outcome than either problem alone, and the only real defence is the habit of checking the pipeline before you touch the model, every time, even when drift feels like the obvious answer.
A cheap habit worth adopting either way: log row counts and null rates for the model's key inputs on every run, alongside the distributions. A distribution shift takes judgement to spot. A source table that suddenly returns half as many rows, or a feature that has gone from two percent null to sixty percent overnight, doesn't. That kind of check won't catch concept drift, but it will save you from ever mistaking a broken upstream job for a philosophical problem about the changing world.
Where to start
If nothing is monitored today, don't start by buying a platform. Start by picking the one or two features that matter most to the model's decision and tracking their distribution weekly, alongside whatever accuracy or proxy signal you can get your hands on given how quickly labels arrive. Set thresholds from your own history rather than a default, and send the first alerts to a person rather than a retrain job.
From there, build the retraining habit before you build the automation: a fixed floor cadence, a shadow test before anything gets promoted, and a standing rule that a sudden drop gets a pipeline check before a drift diagnosis. None of this needs to be complicated to be worth doing. A model that nobody is watching is just a decision made once and then left to go stale, however finished it looked on launch day, and that's a strange thing to trust with anything that matters. If you want an honest read on whether the model you have in production is still the model you think it is, talk to us.
Frequently asked questions
What is the difference between model drift and data drift?
Model drift is the broad decline in a model's accuracy over time, and data drift is one of its two causes: a change in the shape of the inputs while the underlying relationship between inputs and outcome stays the same, such as a retailer's average basket size creeping up because prices rose. The model still knows the right relationship; it's just being handed inputs that increasingly look unlike what it trained on.
What is the difference between model drift and concept drift?
Concept drift is the harder of the two problems: the inputs can look perfectly normal while the relationship between them and the outcome has actually changed, such as fraudsters adapting their tactics once a pattern is known to be caught. Concept drift can't be waited out; it needs new labels and a retrain, because the model is actively learning the wrong thing every time it scores a new case.
What is AI model drift?
Model drift is the gap that opens between a frozen model, which only updates if retrained, and a moving world of changing customer habits, prices, and behaviour. Accuracy erodes without any change to the model's weights, and without monitoring nobody notices until a colleague flags that the numbers look off.
What is model drift in machine learning?
In machine learning, model drift describes a model whose accuracy quietly declines after launch because the data or the relationships it learned no longer match the world it's now scoring. It splits into data drift, where inputs shift shape, and concept drift, where the relationship between inputs and outcome moves, and both need separate detection and fixes.
How can you tell if a model is drifting or if the pipeline is broken?
Look at the shape of the change: genuine drift moves like a slope, degrading gradually over weeks or months as the world changes, while a pipeline bug moves like a cliff, with a metric or a feature's distribution jumping in a single day. Always check whether the inputs still mean what they used to mean, a schema change or a broken join upstream, before assuming the model itself is drifting.