A model scores 98% in testing and everyone relaxes. Then it goes live, the predictions come back wrong, and the team that celebrated last week is now staring at a dashboard that makes no sense. The forecast that looked razor sharp on last year's numbers keeps missing on this quarter's. Most of the time the cause is the same one, and it has a name: overfitting.
Overfitting is what happens when a machine learning model learns its training data too well. It memorises the exact examples it was shown, noise and all, instead of the pattern underneath them. On the data it trained on it looks brilliant. On anything new it falls over. The usual analogy is the student who memorises last year's exam paper: full marks in the mock, lost the moment the questions change.
We build data and AI systems at Shipshape Data, and this is one of the failures we run into most often when a project stalls between a promising notebook and something a business can actually rely on. This guide covers what overfitting is, how to catch it early, what causes it, and the techniques that genuinely stop it, along with its mirror-image problem, underfitting. The aim is practical: things you can check on a model this week, not theory for its own sake.
Why overfitting matters
The whole point of training a model is to generalise: to take what it learned from data you have and apply it to data you do not have yet. Overfitting quietly deletes that ability. The model still produces numbers, still looks confident, but it is answering from memory rather than understanding, and memory is worthless the moment the input changes.
The damage shows up as a performance cliff between training and production. In development the accuracy is excellent. In the real world the same model makes confident predictions that are consistently wrong, because it latched onto quirks in the historical data instead of the relationships that actually drive the outcome. A churn model that memorised which specific customers left last year tells you almost nothing useful about who will leave next year.
A model that has overfit is not learning, it is reciting, and recitation breaks the moment the question changes.
The business cost
That technical failure turns into a commercial one quickly. Decisions get made on predictions that do not hold: marketing budget goes to the wrong audiences, stock forecasts miss, product recommendations point customers at the wrong things. Each bad call chips away at whatever trust the organisation had in the AI programme, and once stakeholders stop believing the outputs, the project rarely gets a second budget.
We have watched teams end up on a treadmill of retrain, test, disappoint, retrain, none of it fixing anything because the real problem was never the amount of training. It was what the model was training on, and how nobody had measured the gap between the lab and the live system until it was already costing money. Overfitting is cheap to catch and expensive to ignore, which is a bad combination if you only look for it after launch.
How to spot it early
Overfitting is far cheaper to fix while a model is still training than after it has shipped, so the habit worth building is watching for it constantly rather than checking once at the end. The clearest signal is a widening gap between how the model does on data it trained on and data it has never seen.
Track both numbers as training progresses. If training accuracy keeps climbing towards 98% while test accuracy stalls at 65% and then starts to slip, that divergence is overfitting happening in front of you. Healthy training moves both lines together. When they split, the model has stopped learning the pattern and started memorising the sample.
There is a more visual version of the same idea, the learning curve. Plot performance against the amount of training data, or the number of training rounds, for both the training and validation sets, and the shape tells the story. Two curves that rise and then sit close together point to a model learning something real. A training curve that races ahead while the validation curve lags behind and eventually bends the wrong way is overfitting drawn out in a single picture. It costs almost nothing to produce and it saves a lot of guessing about what the model is really doing.
Cross-validation
A single train and test split can mislead you, because that one split might just have been an easy draw. K-fold cross-validation gives a steadier read. You divide the data into several subsets, train on some and test on the rest, rotate through every combination, then average the results. Consistent scores across all the folds mean the model is generalising. Scores that swing wildly from fold to fold mean its success depends on which specific examples it happened to see, which is the definition of the problem.
What causes overfitting
Three things cause most of it: a model that is too complex for the data, too little data, and features that carry more noise than signal. They rarely turn up alone. A complex model that would behave with more data, or a simple model drowning in noisy features, can both land you in the same place, which is part of why overfitting is worth understanding rather than just reacting to.
Too much model for the data
A model overfits when it has far more capacity, more parameters, more room to store detail, than the training set can justify. A deep neural network with millions of weights can memorise an entire dataset outright. A decision tree left to grow with no depth limit will keep splitting until every training example sits in its own tiny branch, fitting random wobble as though it were signal. Every extra parameter is another place for the model to record noise it should have ignored.
Not enough data
Small datasets force a model to draw conclusions from thin evidence. Train on 100 rows with 50 features and it will find relationships that are pure coincidence, patterns that exist in your sample and nowhere else. The model has no way to tell a real effect from two things that happened to line up in a handful of records. More data, and more varied data, is often the single most effective fix, because genuine patterns repeat across a large sample while coincidences wash out.
Noisy or redundant features
Feeding a model irrelevant or highly correlated features hands it more ways to cheat. Redundant inputs let it find several routes to the same memorised answer. A feature that correlates with the target purely by accident in the training set becomes a false signal it will happily trust. Careful feature selection and engineering, cutting the inputs that carry noise rather than information, removes those traps before the model can fall into them.
Techniques that actually prevent it
No single trick fixes overfitting. What works is a combination of restraining the model, giving it more to learn from, and validating honestly. These are the ones we reach for most, and roughly what each one buys you.
Early stopping
Train a model long enough and it will eventually start memorising. Early stopping cuts training off at the right moment. You watch validation loss as training runs and stop when it begins rising consistently, even while training loss is still dropping. That rising validation loss is the model beginning to fit noise. Stopping there costs you nothing and saves compute, though it does take some judgement to tell a genuine upturn from ordinary wobble in the curve.
Regularisation
Regularisation adds a penalty to the model's loss function for complexity, nudging it towards simpler patterns that travel better. There are two you will meet constantly. L1 regularisation, or Lasso, pushes some weights all the way to zero, which doubles as feature selection by dropping inputs that were not earning their place. L2 regularisation, or Ridge, shrinks large weights so no single feature can dominate the prediction. Both make the model prefer a smoother, more general fit over an exact one. If you want to go deeper on the wider family of approaches, our notes on AI methods and techniques cover more ground.
Data augmentation
When you cannot get more real data, you can sometimes manufacture useful variety from what you already have. Data augmentation makes modified copies of existing examples: for images, rotating, flipping or adjusting brightness; for text, swapping in synonyms or rephrasing sentences. Each variant is still the same underlying case, so the model has to learn the robust feature rather than the exact pixels or wording. It makes memorising harder and generalising easier, which is precisely the trade you want.
Honest validation splits
Split the data three ways, not two: a training set to fit the model, a validation set to tune hyperparameters and watch for overfitting, and a test set touched exactly once, at the very end, for a final honest read. The discipline that matters here is keeping that test set sealed. Every time you look at it and adjust something in response, you leak a little information back into the model and overfit to the test set itself, which leaves you with no clean measure of anything at all.
Ensembles
Ensembles combine several models so their individual mistakes cancel out. A random forest trains many decision trees on different slices of the data and averages them. Boosting trains models in sequence, each one correcting the errors of the last. Because different models overfit in different directions, pooling them smooths the memorisation away and leaves the shared, genuine pattern behind. It is one of the more dependable ways to buy robustness without hand-tuning every knob yourself.
None of these is a cure on its own, and applying all of them at once is its own kind of mistake, because heavy regularisation stacked on an already simple model can tip you straight into underfitting. The skill is matching the fix to the cause: more data for a data-starved model, a penalty term for an over-eager one, a sealed test set for a team that keeps peeking at its own answers.
Overfitting and underfitting
Overfitting has an opposite, and chasing one too hard drops you straight into the other. Underfitting is a model too simple to capture what is actually going on. Picture a straight line drawn through data that clearly curves: it misses the shape entirely and predicts poorly on both training and test data, because it never had the capacity to learn the pattern in the first place. High bias, low variance, consistently wrong for the same reason every time.
Overfitting is the reverse: low error on the training data, high error the moment anything changes. One model learned too little, the other learned too much of the wrong thing. Your job is the middle ground, where the model captures the real structure and ignores the noise. The tell for that sweet spot is refreshingly simple: similar accuracy on training and test data. A big gap between them means you have overfit. Both numbers poor means you have underfit.
The practical route there is to start simple and add complexity slowly. Begin with a baseline, a linear model or a shallow tree, and only reach for something heavier when the validation numbers earn it. Watch both training and validation metrics at every step, and stop adding complexity the moment test performance stops improving, however good the training score looks. More sophisticated is not the goal. Reliable on data it has never seen is the goal.
Putting this into practice
Knowing the theory and applying it under deadline pressure are two different things. Real projects run on messy data, tight timelines and a business that wanted results yesterday, which is exactly when the temptation to skip validation is strongest and the cost of skipping it highest. Preventing overfitting is less about any one clever method and more about a few habits held consistently.
Start with the data
Most overfitting is decided before modelling even begins, in the state of the data. Clean it first: remove duplicates, handle missing values the same way every time, and deal with the outliers that would otherwise drag the model off course. Split the data before you do any exploration, and keep the test set sealed from that point on. Write down every preprocessing step so you can apply the identical treatment to new data later. A lot of the messy, unstructured data we work through with clients hides exactly the noise that pushes models towards memorising, so this stage more than earns its time.
Build up, not down
Reach for the simplest architecture that could plausibly work and add to it only when the validation numbers justify the addition. A linear model or shallow tree sets a baseline you have to beat before anything fancier is worth considering. Layer in complexity deliberately, checking training and validation metrics at each step, and you will see the exact point where extra capacity stops helping and starts hurting. It saves compute, and it saves you from shipping something more fragile than the problem ever needed.
Watch it after launch
A model that generalised well on the day it launched can drift into trouble later, because the world it was trained on keeps moving. Deploy with monitoring that tracks accuracy over time, set alerts for when predictions slip below an acceptable threshold, and retrain on fresh data that reflects current conditions rather than last year's. Review periodically whether the features still matter and the architecture still suits the problem. Overfitting is not only a training-time issue. A stale model in production is a slower version of the same failure.
Where to start
If you take one thing from all this, make it the gap. The distance between how a model performs on data it has seen and data it has not is the most honest number in the whole process. Split your data properly, cross-validate, keep an eye on that gap, and most overfitting problems will announce themselves long before they ever reach production.
Building models that hold up on real data takes solid data preparation, sensible architecture and monitoring that does not stop at launch, and all of it rests on a data foundation you can trust in the first place. That foundation is the work we do. If your organisation is trying to move AI from something that works in a notebook to something that works in production, talk to us and we will help you find the weak points before they cost you anything.