A fraud model begins operating in January with a ninety per cent success rate. The development team starts a new project because the launch was successful. By September, the same model only catches sixty per cent of fraud cases. The code and deployment settings remain unchanged. The model weights are identical to the original version. The model is less effective because the external environment has changed, and the system cannot detect these changes independently.
Model drift occurs when the accuracy of a machine learning model decreases after deployment despite its code remaining the same. The data the model processes changes as customer behaviour shifts, but a static model cannot adapt to these changes. To identify drift, you must monitor both inputs and outcomes while first ensuring that the data pipeline is functioning correctly.
Shipshape Data builds data and AI systems. One common way these projects fail is through a gradual process that occurs long after launch. While teams monitor the systems initially, they often stop verifying the accuracy of the model outputs over time. Most organisations plan for the development phase, but few prepare for the decline in performance that begins once a model is deployed and the environment around it changes.
This guide explains model drift and the differences between data drift and concept drift. It describes how to identify these issues before they affect users and outlines how to manage daily monitoring and alerting. It also covers how to determine a retraining schedule and how to distinguish between a drifting model and a broken data pipeline.
Why a model that worked at launch stops working
A model is a static representation of patterns found in its training data. It does not update after deployment unless you retrain it. However, the environment where the model operates changes constantly. Customers adopt new habits, suppliers adjust prices, and fraudsters develop new methods. Events like pandemics or economic recessions also shift conditions. Model drift is the discrepancy that develops between the static model and the changing environment. This results in a loss of accuracy even though the model code remains the same.
Some of these issues cannot be prevented. A model trained on three years of stable trading data cannot predict a regulatory change that occurs in the next quarter. Careful engineering during the development phase does not change this fact. The existence of drift is not a mistake.
It is a mistake to treat a launch as the end of a project. A launch is actually the beginning of a continuous task that has not yet been assigned to a specific person.
You should divide this gap into two categories. They result from different causes and require different solutions.
Data drift: the input data changes
Data drift occurs when the input data for a model changes while the relationship between those inputs and the predicted outcome remains constant. For example, a retailer might see an increase in average basket size due to rising prices rather than a change in customer shopping habits. A demographic shift could alter the average age of people applying for loans. An app redesign might change which button users click first, altering a clickstream feature in a way that does not reflect a change in user intent. In these cases, the model still understands the correct relationships, but it is receiving inputs that differ from the data used during its training.
Concept drift: the relationship itself moves
Concept drift is more difficult to manage because inputs may appear standard even when the relationship between those inputs and the predicted outcome has changed. Fraud provides a clear example of this issue. A fraud detection model identifies patterns that were used in the past. Because there is a financial incentive to bypass the model, fraudsters change their methods once they realise a specific pattern is being caught. The data features entering the model may remain identical to how they appeared when the system launched. However, the indicators that previously predicted fraud are no longer effective, and behaviours that once appeared legitimate may now be fraudulent.
Both types are common. If a model did not drift, it would indicate that the environment was no longer changing, which is not a realistic situation.
A recommendation engine displays both issues simultaneously, which explains why retailers retrain their models frequently. Data drift occurs every January when sales end and customer browsing behaviour returns to normal. Concept drift occurs less obviously, such as when a competitor lowers prices in a specific category. In that case, price no longer predicts a purchase as effectively as it did in November. The same model and dashboard are used, but the underlying problems are different.
How to distinguish between the two in practice
This distinction is important because the solutions differ. Data drift is often manageable with the current model for a period of time because the learned relationships remain valid. Concept drift means the model produces incorrect results for every new case it processes, and waiting will not resolve the issue. The model requires new labels and an immediate retraining.
In practice, you rarely receive a clear signal to identify the specific type of change. If a demand forecasting model loses accuracy during a supply shock, the cause could be data drift, where inputs like prices and lead times have shifted. It could also be concept drift, where customer behaviour under scarcity differs from their behaviour during normal supply conditions. Most actual drift events involve a combination of both. You should develop the habit of checking both the input distributions and the model error patterns whenever performance declines. It is rarely possible to label every incident perfectly. However, if a distribution check shows no changes while errors increase, that result is useful. It indicates that concept drift is more likely than data drift, which helps determine your next steps.
How to detect drift before your customers do
Most organisations identify drift through complaints, missed forecasts, or quarterly reviews instead of monitoring systems. This sequence is inefficient, but it occurs frequently because developing effective drift detection requires more effort than most teams anticipate at the start of a project.
Watching the inputs
The population stability index and the Kolmogorov-Smirnov test are the primary methods for comparing current input data against a training distribution for each feature; these are standard tools. The PSI provides a single value per feature to indicate if a distribution has shifted significantly. The KS test determines if two samples likely originate from the same distribution. If you run these tests regularly and plot the results, you can identify data drift early. This allows you to address the issue before it accumulates over several months.
Watching the outcomes
Input checks do not detect concept drift because inputs may remain within expected ranges while the underlying relationships change. To identify this, you must monitor the model's performance metrics over time, such as accuracy, precision, recall, or the specific targets the model was designed to optimise. The difficulty is that ground truth data is often delayed. A loan default model cannot confirm a bad decision for several months, and a churn model only confirms a loss after the customer has left. When labels are slow to arrive, you should monitor proxy signals. These include a decrease in average prediction confidence, a shift in the ratio of positive to negative predictions that lacks a business explanation, or an increasing number of predictions near the decision boundary. This last signal often indicates that the model is becoming less certain than it was previously.
None of this requires custom tooling to start. A cron job that computes PSI on a handful of features and writes the result to a spreadsheet is a genuine monitoring system, and it beats an ambitious platform rollout that stalls in procurement for two quarters. If the workload grows past what a spreadsheet can carry, that's when it's worth comparing dedicated options, and our guide to AWS SageMaker model monitoring covers what setting up drift alerts on a managed platform involves, rather than just what the sales page claims.
Monitoring and alerting systems that users trust
Detection is only effective if someone observes the data and trusts the results. Many drift monitoring systems fail because metrics and dashboards are ignored. If a system produces several false alarms shortly after implementation, users often mute the alerts and stop checking the data.
Thresholds that avoid false alarms
If you use a PSI threshold from a blog post without analysing your own data, the alert will either trigger on normal noise or fail to detect actual shifts. You should set thresholds based on your historical data. Measure how much the metric varies between two typical weeks and set the alert level significantly higher than that variation, rather than choosing a round number that seems reasonable.
Allow a threshold breach to trigger a review instead of an automatic response.
When you set up a new monitor, the first alert should be sent to a person instead of triggering an automated retrain. This is necessary because you do not yet know the typical level of background noise for that monitor.
Choosing the correct metrics for alerts
Set alerts on business metrics whenever possible, in addition to statistical ones. A drop in recall from 92% to 84% represents a specific number of missed fraud cases and a specific amount of revenue exposure. A PSI score of 0.31 is not useful to a reader who does not already understand that metric. Convert statistical signals into the figures the business prioritises. Send these alerts to the individuals responsible for those figures, rather than only the team that built the model. Dedicated MLOps platforms provide these features effectively. These tools still depend on the thresholds and routing rules that a user configures. A well-configured alert on a spreadsheet is more effective than a poorly configured alert on an expensive platform.
Retraining strategy and cadence
There are three general methods for deciding when to retrain a model. Most established systems use a combination of the first two approaches.
Fixed schedule
Retrain your model every week, month, or quarter, depending on how quickly your data changes. This approach is easy to execute and easy to explain to stakeholders. The primary disadvantage is that a fixed schedule treats stable periods and volatile periods the same way. This results in retraining that is occasionally unnecessary or too infrequent to address sudden changes in the model's environment.
Triggered retraining
You should retrain your model when a monitoring threshold is met rather than following a fixed schedule. This method allows you to respond to data drift as it occurs. It is a more efficient approach once your monitoring systems are reliable enough to automate actions. However, this strategy requires trustworthy monitoring. If you implement triggered retraining based on inaccurate alerts, the model will retrain on insignificant data fluctuations.
Shadow testing before you promote anything
Do not deploy a new model to production immediately after a retraining trigger. Run the new model in shadow mode first. This practice allows you to score live traffic alongside the existing model. You can compare their performance on real outcomes before you switch. A retrained model is not always superior to the old one. It has processed more recent data, but that data can be inaccurate. We have seen instances where a retrained model treated three weeks of corrupted upstream data as a valid pattern. This happened because the model was not tested in shadow mode against the model it replaced.
A model in production functions on the assumption that current data will remain similar to the training data. If you do not check this assumption regularly, you are relying on the model without evidence of its continued accuracy.
Most teams should use a fixed schedule for retraining, such as every quarter, to ensure that no model remains unchanged for a full year. Once monitoring systems are reliable, teams can add triggered retraining based on specific performance metrics.
Neither replaces the other.
Is it drift, or is it a broken pipeline
People ask this question frequently, often after a sudden drop in model accuracy. They are usually trying to decide whether the issue requires a data engineer or a data scientist. A sudden, sharp decline is rarely caused by drift. Drift occurs gradually because changes in the real world happen over time. If a model performs well on Tuesday but fails on Wednesday, the cause is almost always a problem in the data pipeline rather than a change in the underlying environment.
The tell is the shape of the change
Actual drift occurs gradually. Performance decreases over weeks or months, following changes in the real world that you could explain through observation. A pipeline bug occurs abruptly. A metric that has been stable for a long period drops in one day, or a feature distribution shifts in a way that real-world changes cannot explain. If a feature that previously ranged from 0 to 100 suddenly contains only zeros, there is a null-handling bug in an upstream ETL job rather than a change in customer behaviour. If a categorical field with eight values suddenly shows a new value in forty per cent of rows, an upstream process changed the schema without notifying the model owners.
You should always check the pipeline first
Before you modify the model, verify that the input data still carries its original meaning.
A source system might have changed the format of a field. A currency conversion could be applying twice. An upstream join may be dropping rows because a key format has changed.
A stable and well-documented data pipeline with real data lineage allows you to perform this check quickly. You can see the origin of a feature and identify any changes that occurred during processing. If a pipeline is not mapped, this same check requires a full day of searching through logs. This difficulty is why teams often claim "the model must be drifting" instead of performing a check. It is easier to make that claim than it is to trace data through six upstream systems.
These two problems can also occur at the same time. A bug in the pipeline may resemble data drift, which can lead a team to retrain the model based on that bug. This process incorporates a broken data feed into the new model rather than fixing the underlying issue. This result is worse than experiencing either problem individually. You can prevent this by checking the pipeline before adjusting the model in every instance, even if data drift appears to be the clear cause.
You should log row counts and null rates for your model's primary inputs during every run, along with the distributions. It requires judgement to identify a shift in distribution. However, you can easily identify when a source table returns half its usual rows or when a feature's null rate increases from two per cent to sixty per cent. These checks do not detect concept drift, but they prevent you from confusing a broken upstream job with changes in real-world data patterns.
Where to start
If you do not have a monitoring system, do not begin by purchasing a platform. Start by selecting the one or two features most important to the model's decisions. Track the distribution of these features every week. Also track accuracy or any available proxy signals, depending on how fast you receive labels. Establish thresholds based on your own historical data rather than using default settings. Send the initial alerts to a person instead of triggering an automated retraining process.
Establish a retraining habit before you implement automation. You should set a fixed schedule for retraining, perform shadow tests before promoting any changes, and investigate the pipeline before diagnosing drift if performance drops suddenly. These steps do not need to be complex to be effective. A model that is not monitored will become outdated, regardless of how complete it seemed at launch. It is not reliable to trust an unmonitored model with important tasks. If you want to know if your production model is still performing as expected, talk to us.
Thinking about building this for your business? Our AI developers will tell you what it depends on, including whether your data is ready to hold it up.
Frequently asked questions
What is the difference between model drift and data drift?
Model drift refers to the general decrease in a model's accuracy as time passes. Data drift is one of the two causes of this decline. It occurs when the distribution of input data changes while the basic relationship between those inputs and the final outcome remains constant. An example is a retailer seeing an increase in average basket size due to rising prices. The model still understands the correct relationship, but the new input data no longer matches the data used during training.
What is the difference between model drift and concept drift?
Concept drift is the more difficult problem. The input data may appear standard even though the relationship between those inputs and the predicted outcomes has shifted. For example, fraudsters often change their methods once they realise a specific pattern is being detected. You cannot wait for concept drift to resolve itself. The model requires new labels and retraining because it learns incorrect information each time it processes a new case.
What is AI model drift?
Model drift is the difference that develops between a model that is no longer updated and a real-world situation where customer habits, prices, and behaviour are changing. A model's accuracy decreases over time even if its settings remain the same. Without monitoring, this decline in accuracy may go unnoticed until someone points out incorrect results.
What is model drift in machine learning?
Model drift occurs when a machine learning model becomes less accurate after deployment because the data or the underlying relationships have changed. This issue is categorised into data drift and concept drift. Data drift happens when the input data changes, while concept drift occurs when the relationship between inputs and outcomes changes. Both types of drift require specific methods for detection and correction.
How can you tell if a model is drifting or if the pipeline is broken?
Observe the pattern of the change. Data drift typically occurs as a gradual slope, where performance or distributions decline over several weeks or months as external conditions shift. A pipeline bug appears as a sudden cliff, causing metrics to drop or distributions to shift abruptly in a single day. Before concluding that the model is drifting, verify that the inputs remain consistent. Check for upstream schema changes or broken joins that may have altered the meaning of the data.