The model finishes training at 2am on a Tuesday. The accuracy numbers look good, everyone claps in the stand-up, and the team moves on to the next thing. Three weeks later the same model is live behind an API, a customer clicks "get a quote," and nothing happens for four seconds. Nobody sized that. Training happened once, on a machine somebody rented for a weekend. This has to happen every single time someone asks, and it has a bill attached to it that nobody wrote down.
Inference is what happens when a trained model is actually used: new data goes in, the model runs its learned weights over it, and a prediction comes out. No labels, no gradient updates, no learning. The model that does inference is frozen. It is the same set of numbers every time, applied to whatever you hand it, and the answer it gives is only as good as what got baked in during training. Training is where a model earns its judgement. Inference is where that judgement gets used, over and over, at whatever pace the business needs it.
We build data and AI systems at Shipshape Data, and inference is the part that gets designed as an afterthought more often than it should. Teams spend months on the training run, argue over architectures and features, then bolt the model onto an endpoint the week before launch without asking how many requests a second it needs to handle. This guide covers what happens at inference time, how it differs from training, where the cost and the latency come from, when to run inference in batches instead of live, and what running it in production actually involves once the demo is over.
What actually happens when a model makes a prediction
Strip away the branding and inference is arithmetic. The model is a fixed set of numbers, its weights, arranged in a structure decided at training time. A request comes in, gets converted into the same shape of numbers the model was trained on, and those numbers get multiplied and added through the model's layers until something comes out the other end: a class, a score, a next word, a probability.
The forward pass, and nothing else
This is called a forward pass, and it runs in one direction only. Data goes in, moves through the layers in order, and a result comes out. Training also runs forward passes, but it pairs each one with a backward pass that compares the output to a known answer and nudges the weights to be slightly less wrong next time. Inference drops the backward pass entirely. There is no known answer to compare against, because if there were, you would not need the model. The weights stay exactly as they were left at the end of training, and they do not change no matter how many requests arrive or how obviously wrong some of the answers turn out to be.
Same weights, different question every time
That fixed-ness is the whole point and also the whole risk. A model deployed in January answers a query in July using exactly the January weights, even though the customers and products it describes have moved on since then. Inference cannot notice that drift on its own. It runs the arithmetic it was given and returns an answer with the same confidence whether the input looks like anything it saw in training or nothing like it at all. Watching for that gap is a job for model monitoring in production, not for the inference step itself, which has no opinion about whether it should still be trusted.
Training and inference are different jobs
Treating training and inference as the same problem with different data is a common mistake, and it explains a lot of the surprises teams hit at launch. They optimise for entirely different things.
What training optimises for
Training cares about accuracy above almost everything else. It runs offline, on a schedule nobody but the data science team is watching, and it can take hours or days without anyone outside the project noticing. Cost matters, but it is a one-off cost, paid once per run. If a training job takes six hours instead of four, that is mildly annoying. Nobody's product breaks.
What inference optimises for
Inference cares about speed, cost per request and reliability, and it runs constantly rather than occasionally. Training happens when someone kicks it off. Inference happens every time a user does something, which for a live product can mean thousands of times a minute, and the bill for those milliseconds is paid on every single request rather than once. This is also why the compute shape looks different: training wants raw throughput on the biggest hardware you can justify, inference wants hardware sized correctly for the traffic, running continuously, and idling expensively if you got the sizing wrong.
A model can be trained cheaply and still be expensive to run, or trained expensively and still be cheap to serve. The two numbers barely correlate. Teams that only track training cost are tracking the smaller of the two bills.
Where the cost of inference actually comes from
Ask most teams what inference costs and you get a shrug, or a cloud bill nobody has broken down by workload. The cost is real and it comes from three places: compute, memory and the shape of the traffic itself.
Compute per request
Every prediction runs the model's arithmetic again, which means every prediction pays for compute again. A small model doing a simple classification might cost a fraction of a penny per call. A large language model generating a long answer token by token costs meaningfully more, because each token is effectively another small inference pass built on everything generated so far. Multiply either number by request volume and the totals stop looking trivial once a feature that seemed cheap in a demo gets used by everyone rather than a handful of testers.
Memory and the hardware you actually need
A model has to sit in memory before it can answer anything, and for anything beyond a small model that means specialised hardware: a GPU, or sometimes a purpose-built inference chip, holding the weights whether or not a request is currently arriving. That hardware is not free while it waits. Idle GPU time is one of the quiet ways inference budgets balloon, because provisioning for peak traffic means paying for capacity that sits mostly unused outside the busy hours, and provisioning for average traffic means the busy hours are where things fall over.
Batching, and its limits
One genuine lever is batching several requests together so the hardware processes them in a single pass instead of one at a time, which uses the same GPU far more efficiently. It is a real saving and worth building into any serving setup that has the traffic to support it. The catch is that batching trades cost for latency: a request now waits for the batch to fill, or for a timeout, before it gets processed at all, so the saving on your cloud bill shows up as a delay on somebody's screen. Whether that trade is worth making depends entirely on what the answer is for.
Latency and throughput pull in different directions
Two numbers describe how an inference system performs, and they are not the same number wearing different names. Confusing them is how a system gets built that is fast in the dashboard and slow for the person waiting on an answer.
Latency is what one person feels
Latency is the time between a single request going in and its answer coming out. It is what a customer experiences, and it is what gets noticed the moment it slips. A chatbot that takes four seconds to reply feels broken even if it eventually gives a good answer, because people expect conversation to move at conversation speed. Latency is driven by model size, hardware, network hops and how much work happens before the model even sees the request, and it is measured properly at the tail, the slowest one percent of requests, rather than the average, because the average hides exactly the failures a customer will actually hit.
Throughput is what the system handles
Throughput is the total number of requests a system can process in a given stretch of time, and it is what an engineering team watches on a graph rather than what any single user feels directly. A system can have excellent throughput and still deliver a miserable individual experience, if it achieves that throughput by queuing requests to fill a batch. It can also have modest throughput and feel snappy to every single user, if it is sized generously and never has to make anyone wait.
A system tuned purely for throughput will happily keep everyone waiting a little, as long as it processes an impressive number of them per second.
The trade-off has to be a decision, not an accident
Optimising one usually costs you some of the other, and which to favour is a business decision dressed up as an engineering one. A fraud check that runs before a payment clears needs latency measured in tens of milliseconds, whatever that costs in wasted capacity. A nightly model scoring every customer for churn risk needs throughput across millions of rows and does not care if any one row takes half a second, because nobody is staring at a screen waiting for row 40,000. Building the wrong one wastes money in the first case and loses customers in the second.
Batch inference vs real-time inference
Most of the decisions above collapse into one choice made early and rarely revisited: does this prediction need to happen the moment it is asked for, or can it happen on a schedule and be ready when it is needed?
Batch inference
Batch inference runs predictions on a large set of data all at once, usually on a schedule: nightly, hourly, weekly, whatever the use case tolerates. Nobody is waiting on any individual prediction while it runs, which means the system can be tuned purely for throughput and cost, with generous batching and cheaper hardware that would be unusable for a live product. A churn score, a demand forecast, a set of product recommendations refreshed overnight, all of these are naturally batch. The honest trade is staleness: the prediction is only as current as the last run, and if something changes in the gap between runs, the system will not know until the next scheduled pass.
Real-time inference
Real-time inference answers a request the moment it arrives, typically within a service level measured in milliseconds rather than minutes. A fraud check on a live transaction, a search ranking, a chatbot reply, a recommendation generated the instant someone lands on a page: all of these need an answer now, because the value of the prediction depends on it arriving inside the interaction it belongs to. The cost is that the whole serving stack has to be provisioned for the worst moment, not the average one, and that provisioning costs money through every quiet hour in between.
Picking the wrong one is the actual failure mode
Most teams do not fail by picking a bad model. They fail by building real-time infrastructure for a prediction that could have run in a nightly batch, or by building a batch pipeline for a case that quietly needed to be live, and only finding out when the product team asks why recommendations are a day old. The question worth asking before any engineering starts is simple: what changes if this answer arrives an hour later instead of instantly? If the honest answer is nothing, batch it. If the feature stops making sense without an instant answer, it was real-time from the start.
What running inference in production actually involves
Getting a model to produce one correct prediction on a laptop is a small part of the job. Getting it to do that reliably, for thousands of different inputs, at whatever hour someone happens to need it, is most of the actual work.
Serving infrastructure
A model in production sits behind a serving layer: something that loads the weights, exposes an endpoint, handles concurrent requests, and scales instances up when traffic rises and down when it falls, ideally before anyone notices either direction. This is the layer that turns a trained artefact into something a product can call, and building it badly is how a model that worked perfectly in a notebook becomes unreliable the moment real traffic hits it. Our overview of model deployment covers that handover in more depth than fits here.
Versioning, and knowing which model answered
Models get retrained, tweaked and replaced, and production needs to know which version answered which request, because the day something goes wrong you will need to trace the bad prediction back to a specific set of weights, not a vague sense that "the model" did something odd. Rolling out a new version without a way to compare it against the old one, or roll back quickly if it behaves worse live than it did in testing, is how a routine update turns into an incident.
Monitoring what actually happens after launch
A model that performed well in testing can degrade quietly once it meets real traffic, through drift in the incoming data or edge cases nobody thought to test. None of that is visible unless somebody is watching accuracy, latency and error rates on the live system, rather than only the numbers from the training report. Overfitting, which we cover elsewhere, is a training-time failure you can catch before launch. Inference-time drift only shows up after the model has been answering real questions for a while, which is what makes it easy to miss.
Where inference quietly breaks
Most inference failures are not dramatic. The model does not crash, it just gets slowly worse in ways that are easy to miss until someone finally goes looking.
Cold starts
An instance that scaled down during a quiet period has to load the model back into memory before it can answer the next request, and that load can take seconds, landing as a painfully slow response for whoever arrives first after the quiet spell. Keeping a warm instance running avoids it, at the cost of paying for capacity that sits idle. There is no free version of this trade, only which side you would rather explain.
Averages that hide the actual complaint
A system that answers in 80 milliseconds on average and 6 seconds for one request in five hundred looks fine on a dashboard built around the mean. That one in five hundred is the request a real person hit, and nobody experiences an average, only their own request. Teams that track only average latency are, in effect, choosing not to know about their worst experiences, which are usually the ones that end up in a complaint.
Where to start
Work out which of your predictions genuinely need to be instant and which could run on a schedule, before any infrastructure gets built, because that single decision shapes almost everything downstream: the hardware, the cost, the team's on-call burden. Most organisations have fewer true real-time cases than they assumed, and moving even one of them to batch can quietly cut a serving bill.
Then measure before you optimise. Time an actual request end to end, at the tail rather than the average, and price out what a thousand of them cost on the hardware you plan to use. Most nasty surprises in production inference are things nobody measured until a customer complained, and most of them were measurable in an afternoon.
Inference is the part of an AI system that a customer actually meets, and it deserves the same rigour as the model itself. If you are moving something from a promising demo towards a system that has to hold up under real traffic and a real bill, talk to us and we will help you size it honestly before it is live and too late to change.