For six years a retailer's fraud team ran on the same rulebook: decline any order over £500 shipped to a freight forwarder, flag any account used within an hour of being created, block any card used from three countries in a day. The rules worked, and then one quarter they stopped. A new pattern of fraud came through underneath them: small orders, established accounts, one country, spread patiently across months. Nobody had written a rule for a pattern they had never seen, because that is not how rules work. You can only write down what you already know to look for.
Machine learning is what you get when you stop writing that rulebook and give the system a different instruction instead: don't tell me the pattern, show me thousands of examples of it and work it out yourself. Instead of an engineer typing "if amount over £500 and destination is a freight forwarder, decline", you hand the system years of transactions already labelled fraud or not fraud, and it finds the combination of signals, sometimes hundreds of them, that actually separates the two. Nobody needs to be able to state the rule in advance. The system infers it from the data in front of it and applies whatever it found to transactions nobody has seen yet.
We build data and AI systems for clients at Shipshape Data, and the mistake we run into most is rarely a bad algorithm. It is aiming machine learning at a problem a spreadsheet formula would have solved just as well, or the opposite, sticking with a rules engine for years after the pattern it was chasing had already outgrown it. Both mistakes trace back to the same gap: not being clear on what machine learning actually does differently from the rules it is meant to replace.
This guide covers the three main ways a system learns, supervised, unsupervised and reinforcement, what actually happens during training from raw data to a model somebody can rely on, the algorithm families you meet again and again once you start looking, and how to tell whether the problem in front of you is genuinely one machine learning suits, or one it will quietly make worse.
Why rules stop working
A rules engine is a list of conditions somebody wrote down because they had already seen the pattern. That works fine while the world keeps producing the pattern you anticipated. It falls over the moment reality drifts, because a rule only fires on the exact shape it was written for, and fraud, customer behaviour and machinery wear do not hold still to be nice about it.
Rules are explicit, learning is inferred
The honest way to describe the difference: a rules engine encodes a human's understanding of a pattern. A trained model encodes a pattern the human never had to understand at all. You feed it examples, it adjusts its internal parameters until its predictions match the labelled outcomes closely enough, and what it ends up with is a set of weightings across dozens or hundreds of variables that nobody sat down and reasoned through. That is the entire trick, and it is also the entire cost.
A rules engine tells you exactly why it declined the order. A trained model tells you it declined the order, and the reason lives somewhere across a few hundred weighted signals that nobody wrote down and nobody fully reads back.
The trade nobody mentions upfront
This is the part sales conversations tend to skip. Moving from rules to learning buys you patterns nobody could have specified by hand, and it costs you the ability to point at a single line and say "that's why". Some industries can live with that trade easily. Others, credit decisions and anything touching regulated advice among them, cannot wave it away, and the honest answer is sometimes that a simpler, explainable model is worth more than a marginally more accurate one nobody can account for.
Three ways a system can learn
Machine learning is not one technique wearing different clothes. There are three genuinely different ways to set up the learning problem, and picking the wrong one wastes months before anyone notices the model was never going to work.
Supervised learning: learning from labelled answers
Supervised learning is the workhorse, and probably what most people picture when they hear the term. You give the system input and the correct answer for each example, thousands of past loan applications each already marked approved or defaulted, thousands of emails already marked spam or not, and it learns the mapping between the two. Once trained, it applies that mapping to new cases where the answer is not yet known. It needs labelled data, which is the expensive part, and it is the closest thing in this field to a reliable, well-understood starting point. Our fuller guide on supervised learning goes through the main algorithms and where each earns its keep.
Unsupervised learning: finding structure nobody labelled
Unsupervised learning has no answer key. You hand it a pile of data with no labels and ask it to find the structure sitting inside, which customers cluster together by behaviour, which transactions look unusual against the rest, which variables move together and which are actually redundant. Nobody tells the model what a segment should look like beforehand. It surfaces one from the shape of the data itself, and it is often the honest first step when you suspect there is structure in a dataset but genuinely do not know what it is yet. We cover the main techniques, from clustering to dimensionality reduction, in the unsupervised learning guide.
Reinforcement learning: learning from consequences
Reinforcement learning is the odd one out, and the one most overhyped relative to how often it actually applies to a business problem. There is no fixed dataset of correct answers. Instead an agent takes actions inside an environment, gets a reward or a penalty depending on the outcome, and adjusts its behaviour to chase more reward over time. It is how systems learn to play games better than any human, and it does real work in robotics, logistics routing and some pricing problems. It is a poor fit for most enterprise data problems, because it needs an environment you can simulate cheaply and repeatedly, and most businesses do not have one lying around. It genuinely earns its place in a narrow set of use cases, and gets pitched at plenty more than that.
What training actually looks like, step by step
Strip away the vocabulary and training a model is a fairly plain sequence of steps, repeated with small variations depending on the problem. Most of the effort sits earlier than people expect, and none of it is the part that gets demoed.
Getting the data into shape
This is the unglamorous majority of the work, and it is also where most projects quietly stall. Raw data arrives duplicated, mislabelled, missing half its fields and inconsistent about what a customer even is across two systems. None of that gets fixed by a cleverer algorithm. It gets fixed by going through the data by hand, deciding what to do with the gaps, and being consistent about it, which is precisely why weak data quality shows up as a modelling problem months later when it was actually a data problem from day one. Features then get engineered from the cleaned data, turning raw fields into the signals a model can actually use: an age from a date of birth, a rolling average from a stream of transactions, a distance from two postcodes.
Splitting, fitting and testing honestly
The data gets split, usually into a training set the model learns from, a validation set used to tune choices along the way, and a test set touched exactly once at the end for an honest read. The model then fits itself to the training data, adjusting its internal parameters to reduce the gap between its predictions and the known answers. This is where the discipline matters most, because a model that performs brilliantly on the data it trained on and poorly on anything new has not learned the pattern at all, it has memorised the examples. We have written at length about that specific failure in our guide to overfitting, because it is the single most common reason a promising model falls apart the moment it meets real data.
Deployment is not the finish line
A model that scores well in testing still has to survive contact with production data, which behaves differently the moment real users, real seasons and real edge cases start hitting it. Deployment needs monitoring built in from the start, not bolted on once something has already gone quietly wrong. The world the model was trained on keeps moving, customers change, prices change, fraud patterns adapt specifically because a model is now watching for the old ones, and a model left untouched slowly drifts out of step with the reality it is scoring. Retraining on a schedule, or in response to a measured drop in accuracy, is part of running the thing, not an optional extra.
The algorithm families you'll meet
Underneath the three learning styles sits a smaller set of algorithm families that keep reappearing, in different combinations, across almost every practical project. Knowing the shape of each saves you from either underselling a simple problem or overbuilding a straightforward one.
Linear models and decision trees
Linear and logistic regression are the plainest tools in the box: they fit a straight line, or its probability equivalent, through the data and are genuinely hard to beat for problems where the relationship really is roughly linear. They are fast, explainable, and the correct starting point more often than people assume. Decision trees split the data on one question at a time, is the order value over £500, is the customer new, building a flowchart that a person can read end to end. Both are underrated. A well-built linear model that a compliance team can actually interrogate sometimes beats a fractionally more accurate model nobody can explain.
Ensembles: many weak models outvoting one
A single decision tree overfits easily, chasing noise until every training example sits in its own tiny branch. Ensembles fix this by combining many trees rather than trusting one. Random forests train a large number of trees on different slices of the data and average their votes. Gradient boosting trains trees in sequence, each one correcting the mistakes of the last. Both routinely outperform a lone tree on messy, real-world tabular data, and they are the default choice for a huge share of the structured business problems, churn, pricing, credit risk, that get solved with machine learning day to day.
Neural networks
Neural networks stack layers of simple mathematical units that each combine and reweight signals from the layer before, and given enough of them, enough data and enough compute, they can approximate almost any pattern. That power is exactly why they excel at images, audio and language, tasks where the raw input is unstructured and no set of hand-built features would capture what matters. It is also why they are frequently the wrong choice for a straightforward spreadsheet of customer records, where a much simpler model would perform just as well with a fraction of the data, compute and explainability cost.
Spotting a problem machine learning actually suits
Not every prediction problem needs a model, and treating every question as a machine learning question is one of the more expensive habits a data team can pick up. A handful of honest checks separates the genuine candidates from the ones that will waste a quarter.
The questions worth asking before you start
Is there a real pattern to learn, one that repeats often enough for examples of it to exist in your data? A one-off event has nothing for a model to generalise from. Do you have enough labelled history to learn from, and is it labelled honestly rather than by whichever proxy was easiest to log? Would a human expert, given the same inputs, actually struggle to write the rule down themselves? If a colleague could jot the logic on a whiteboard in ten minutes, you probably need that whiteboard, not a model. And can you tolerate being wrong sometimes, because every model is wrong some proportion of the time, and a use case with zero tolerance for error is not a machine learning use case yet, whatever the roadmap says.
Fraud, churn, demand forecasting, predictive maintenance, image and text classification: these keep showing up as good fits because they share the same shape. A real pattern, enough history, a rule too complex for a person to hand-write, and a business that can absorb the model being wrong on the cases at the margin.
Where machine learning is the wrong answer
The counterpart to spotting a good fit is admitting a bad one, and this is the section most vendors skip. Sometimes the honest answer is that you do not need this.
If the logic really is a handful of conditions and they rarely change, a rules engine will do the job more cheaply, more transparently, and with far less to maintain. If your data is too thin, too recent, or too inconsistently labelled, a model trained on it will confidently learn the noise rather than the signal, which is arguably worse than having no model at all, because it looks like an answer. And if the underlying pattern moves faster than you can retrain, a weather-dependent demand spike nobody had data for last year, a model built on history will keep predicting last year's world with total confidence right up until it is visibly, embarrassingly wrong.
We have sat in enough scoping calls to recognise the shape of a project reaching for machine learning because it sounds more serious than the alternative. Sometimes the right recommendation is a dashboard and a documented rule, and saying so is worth more to the client than pretending otherwise.
Where to start
Start with the problem, not the technique. Write down the decision the model is meant to change, the data you already have to learn from, and whether a person could describe the pattern in a sentence. If they can, you probably do not need a model yet. If they cannot, and the history exists to learn from, you have a genuine candidate.
Then get the unglamorous groundwork right before anything else: clean, consistently labelled data, a proper split between training and testing, and monitoring that starts on day one rather than after something has gone wrong. None of that is exciting, and all of it decides whether the model that comes out the other end is something the business can actually rely on.
If you are trying to work out whether a problem you are looking at genuinely suits machine learning, or you have a model already built and are not sure it was ever the right tool, talk to us.