A fraud model goes live in January catching nine cases in ten, and the team that built it moves on to the next thing, because that is what a good launch looks like. By September the same model is catching six in ten. Nobody touched the code. Nobody redeployed it. The weights are exactly what they were the day it shipped. What changed is the world the model is now looking at, and the model has no way of noticing that on its own.
We build data and AI systems at Shipshape Data, and this is one of the quieter ways an AI project goes wrong. Not at launch, when everyone is watching, but months later, when the dashboard still says the model is running and nobody is checking whether it is still right. Teams plan for the build. Fewer plan for the slow, unannounced decay that starts the moment the model stops learning and the world keeps moving.
This guide covers what model drift actually is, the difference between data drift and concept drift, how to catch either one before customers do, what a monitoring and alerting setup should look like day to day, how to think about retraining cadence, and the question we get asked most often on this topic: how do you tell a drifting model apart from a pipeline that has quietly broken.
Why a model that worked at launch stops working
A model is a frozen snapshot of a pattern that existed in the data it was trained on. Nothing about it updates once it is deployed, unless you explicitly retrain it. The world it is scoring, on the other hand, keeps moving: customers change habits, suppliers change prices, fraudsters change tactics, a pandemic happens, an economy tips into a recession. Model drift is the gap this opens up between the frozen pattern and the moving target, and it shows up as accuracy that erodes without any change to the model itself.
Some of this is genuinely unavoidable. A model trained on three years of stable trading conditions has no way to anticipate a regulatory change that lands next quarter, and no amount of careful engineering at build time changes that. Drift itself is not the mistake. The mistake is treating a launch as the finish line, when a launch is closer to the starting gun for an ongoing job that nobody actually assigned to anyone.
It helps to split the gap into two different things, because they are caused by different problems and fixed in different ways.
Data drift: the inputs change shape
Data drift is a change in the inputs the model sees, while the underlying relationship between those inputs and the outcome stays the same. A retailer's average basket size creeps up because prices rose, not because shopping habits changed. A demographic shift moves the average age of loan applicants. An app redesign changes which button people click first, which changes the shape of a clickstream feature that has nothing to do with intent. The model still knows the right relationship. It is just being handed inputs that look increasingly unlike what it trained on.
Concept drift: the relationship itself moves
Concept drift is nastier, because the inputs can look perfectly normal while the relationship between them and the outcome has changed underneath. The clearest example is fraud. A fraud detection model learns the patterns fraudsters used last year. Fraudsters do not stand still, because there is money on the other side of the model, and once a pattern is known to be caught, it stops being used. The features feeding the model can look exactly as they did at launch. What used to predict fraud no longer does, and something that used to look innocent now is not.
Both kinds are normal. A model that never drifted would mean the world had stopped changing, which is not the world anyone actually operates in.
A recommendation engine shows both at once, which is part of why retailers get so used to retraining. Data drift arrives every January when the sale ends and browsing behaviour resets. Concept drift arrives more quietly, when a competitor undercuts a category and price stops being the thing that predicts a purchase the way it did in November. Same model, same dashboard, two different diseases.
Telling the two apart in practice
The distinction matters because the fix is different. Data drift is often survivable with the existing model, at least for a while, because the relationship it learned is still true. Concept drift means the model is actively learning the wrong thing every time it scores a new case, and no amount of waiting fixes that. It needs new labels and a retrain, sooner rather than later.
In practice you rarely get a clean signal telling you which one you are looking at. A demand forecasting model that starts missing during a supply shock could be either: the inputs shifted (data drift, prices and lead times moved) or the relationship shifted (concept drift, customers now behave differently under scarcity than they did when supply was normal). Most real drift events are a mix. The useful habit is not trying to label every incident perfectly, it is checking both the input distributions and the model's error pattern whenever performance slips, because a distribution check that comes back clean while errors climb is itself informative: it points you towards concept drift and away from data drift, which changes what you do next.
How to detect drift before your customers do
The honest starting point is that most organisations find out about drift from a complaint, a missed forecast, or a quarterly review, rather than from a monitoring system. That is backwards, and it is also completely normal, because building drift detection well takes more than most teams expect on day one.
Watching the inputs
Population stability index and the Kolmogorov-Smirnov test are the two workhorses for comparing a current batch of inputs against the training distribution, feature by feature. Neither is exotic. PSI gives you a single number per feature that flags when a distribution has shifted meaningfully; the KS test tells you whether two samples plausibly came from the same distribution at all. Run either on a schedule, plot the trend, and you catch data drift while it is still small, rather than after it has compounded for six months.
Watching the outcomes
Input checks miss concept drift, because the inputs can look fine while the relationship has moved. For that you need to track the model's actual performance over time: accuracy, precision, recall, whatever the model was built to optimise. The complication is that ground truth often lags. A loan default model does not know if a loan was a bad decision for months. A churn model does not know if someone churned until they have actually gone. When labels are slow, watch proxy signals instead: prediction confidence drifting lower on average, the ratio of positive to negative predictions shifting when nothing in the business explains why, or a rising share of predictions sitting right on the decision boundary, which is often the first visible sign that the model is less sure of itself than it used to be.
None of this requires custom tooling to start. A cron job that computes PSI on a handful of features and writes the result to a spreadsheet is a genuine monitoring system, and it beats an ambitious platform rollout that stalls in procurement for two quarters. If the workload grows past what a spreadsheet can carry, that is when it is worth comparing dedicated options, and our guide to AWS SageMaker model monitoring covers what setting up drift alerts on a managed platform actually involves, rather than just what the sales page claims.
Monitoring and alerting that people actually trust
Detection only works if somebody sees it and believes it. This is where a lot of drift monitoring quietly fails: the metric exists, the dashboard exists, and nobody looks at it, because it fired three false alarms in its first month and got muted.
Thresholds that do not cry wolf
A PSI threshold copied from a blog post, applied without checking what normal variation looks like for your own data, will either alert constantly on noise or stay silent through a real shift. Set thresholds from your own history: look at how much the metric moves between two ordinary weeks, and set the alert meaningfully above that, not at a round number that felt sensible. Then let a threshold breach earn attention rather than automatic action. The first alert on a new monitor should go to a person, not trigger a retrain, because you are still learning what your own noise floor looks like.
Alerting on the right thing
Alert on the business metric where you can, not only the statistical one. A recall drop from 92% to 84% means something concrete: a specific number of missed fraud cases, a specific revenue exposure. A PSI score of 0.31 means nothing to the person reading the alert unless they already know what PSI is. Translate the statistical signal into the number the business actually cares about, and route it to whoever owns that number rather than only the team that built the model. Dedicated MLOps platforms do this reasonably well out of the box. They are still only as good as the thresholds and the routing someone configures on top, and a well-configured alert on a spreadsheet beats a badly configured one on an expensive platform.
Retraining strategy and cadence
There are three broad ways to decide when to retrain, and most mature setups end up running some blend of the first two.
Fixed schedule
Retrain weekly, monthly, quarterly, whatever cadence matches how fast your domain moves. It is simple to run and simple to explain to a stakeholder. Its weakness is obvious once you say it out loud: a fixed schedule retrains on a calm month exactly as often as a month where the ground shifted under the model, which means it is sometimes wasteful and sometimes too slow.
Triggered retraining
Retrain when a monitoring threshold fires, rather than on a clock. This responds to actual drift rather than the passage of time, and it is the more efficient approach once your monitoring is trustworthy enough to act on. The catch is right there in that sentence: it only works if the monitoring is trustworthy, which is the entire previous section. Triggered retraining bolted onto noisy alerts just means retraining on noise.
Shadow testing before you promote anything
Whichever cadence triggers a retrain, do not push the new model straight to production. Run it in shadow, scoring live traffic silently alongside the current model, and compare the two on real outcomes before the new one takes over. A retrained model is not automatically better. It has just seen more recent data, and more recent data is not always cleaner data. We have seen a retrain go out that quietly encoded three weeks of a broken upstream feed as a new pattern worth learning, because nobody shadow tested it against the model it was replacing.
A model in production is a bet that the world will stay roughly the shape it was on training day, and every month that bet goes unchecked is a month you are trusting it on faith alone.
For most teams the practical answer is a fixed floor (retrain at least quarterly, so nothing goes a year untouched) with triggered retraining layered on top once the monitoring earns that trust. Neither replaces the other.
Is it drift, or is it a broken pipeline
This is the question we get asked most, usually by someone who has just watched accuracy fall off a cliff overnight and is trying to work out whether to call a data engineer or a data scientist. The honest answer is that a sudden, sharp drop is rarely drift. Drift is gradual almost by definition, because the world changes gradually. A model that was fine on Tuesday and badly wrong on Wednesday almost always has a pipeline problem behind it, not a philosophical one.
The tell is the shape of the change
Genuine drift moves like a slope: performance degrades over weeks or months, tracking a real shift in the world you could plausibly explain if you went and looked. A pipeline bug moves like a cliff: a metric that was stable for months falls off in a single day, or a feature's distribution jumps in a way no real-world change explains. If a feature that used to range from 0 to 100 suddenly shows nothing but zeros, that is a null-handling bug in an upstream ETL job, not customers suddenly behaving differently. If a categorical field that had eight sensible values overnight has one new value showing up in forty percent of rows, something upstream changed its schema and nobody told the model.
Check the pipeline first, always
Before touching the model, check whether the inputs it is receiving even mean what they used to mean. Has a source system changed a field's format. Has a currency conversion started applying twice. Has a join upstream started silently dropping rows because a key format changed. A stable, well-documented data pipeline with real data lineage makes this check quick: you can see exactly where a feature came from and what changed along the way. A pipeline nobody has mapped turns the same check into a day of grepping through logs, which is usually the actual reason teams reach for "the model must be drifting" without checking first, because it is genuinely easier to say than to trace six upstream systems.
The two problems can also compound. A pipeline bug can look enough like data drift that a team retrains around it, baking a broken feed permanently into the new model instead of fixing the feed. That is a worse outcome than either problem alone, and the only real defence is the habit of checking the pipeline before you touch the model, every time, even when drift feels like the obvious answer.
A cheap habit worth adopting either way: log row counts and null rates for the model's key inputs on every run, alongside the distributions. A distribution shift takes judgement to spot. A source table that suddenly returns half as many rows, or a feature that has gone from two percent null to sixty percent overnight, does not. That kind of check will not catch concept drift, but it will save you from ever mistaking a broken upstream job for a philosophical problem about the changing world.
Where to start
If nothing is monitored today, do not start by buying a platform. Start by picking the one or two features that matter most to the model's decision and tracking their distribution weekly, alongside whatever accuracy or proxy signal you can get your hands on given how quickly labels arrive. Set thresholds from your own history rather than a default, and send the first alerts to a person rather than a retrain job.
From there, build the retraining habit before you build the automation: a fixed floor cadence, a shadow test before anything gets promoted, and a standing rule that a sudden drop gets a pipeline check before a drift diagnosis. None of this needs to be complicated to be worth doing. A model that nobody is watching is just a decision made once and then left to go stale, however finished it looked on launch day, and that is a strange thing to trust with anything that matters. If you want an honest read on whether the model you have in production is still the model you think it is, talk to us.