The model completed its training at 2am on a Tuesday. The accuracy figures appeared satisfactory, the team applauded during the stand-up, and then they proceeded to the next task. Three weeks afterwards the identical model was deployed behind an API, a user clicked "get a quote," and there was a four-second pause with no response. No one estimated that. The training occurred a single time, on a machine that someone rented for a weekend. This must be performed each time a request is made, and it carries a cost that no one recorded.
In AI, inference refers to the act of employing a trained model. The model uses its fixed weights on fresh data to generate a prediction. This operation does not entail any learning or modifications to the model. Training defines the model's logic, whereas inference applies that logic to particular tasks at the speed and expense demanded by the business.
Inference takes place when an individual uses a trained model. The model takes in new data, applies its learned weights, and yields a prediction. This procedure does not involve labels, gradient updates, or any learning. The model employed for inference is frozen. It uses the same set of numbers for each input. The output quality relies completely on the information set during the training phase. Training is the process by which a model forms its judgement. Inference is the stage where that judgement is repeatedly applied at the speed the business needs.
Shipshape Data constructs data and AI systems. Inference is frequently treated as an afterthought. Teams often devote months to training, architectures, and features, yet they attach the model to an endpoint just before launch without estimating the needed requests per second. This guide describes the inference process and how it differs from training. It also discusses the sources of cost and latency, the decision between batch and live inference, and the requirements for operating models in production after the initial demo.
What happens when a model makes a prediction
If you remove the branding, inference is simply arithmetic. The model consists of a fixed set of numbers-its weights-organised in a structure determined during training. A request arrives, is transformed into the same shape of numbers the model was trained on, and those numbers are multiplied and added through the model's layers until an output emerges: a class, a score, a next word, or a probability.
Only the forward pass
This operation is a forward pass that proceeds in a single direction. Data enters the system, moves through the layers sequentially, and yields a result. During training, the system carries out forward passes and then follows each with a backward pass. The backward pass compares the output to a known answer and modifies the weights to lower future errors. Inference does not involve a backward pass. No known answer exists for comparison, because the model would be unnecessary if the answer were already known. The weights stay fixed at the values set during training. They do not change regardless of how many requests are made or how accurate the results are.
Same weights, different question every time
Fixed weights constitute both the main function and the main risk of these models. A model launched in January will employ those identical January weights to answer a query in July, even if customers and products have changed in the interim. The inference process cannot detect this drift. It carries out the programmed arithmetic and delivers an answer with the same confidence level whether or not the input resembles the training data. Identifying this gap falls to model monitoring in production. The inference step itself does not assess whether its outputs remain reliable.
Training and inference are separate processes
Teams frequently confuse training and inference as the same problem with different data. This confusion leads to many of the unexpected issues that arise during a launch. These two processes are optimised for different objectives.
What training optimises for
Training places accuracy above other considerations. The process runs offline following a schedule overseen by the data science team. It may run for hours or days without impacting external users. Although cost matters, it is a one-time expense for each run.
A training job lasting six hours rather than four is a minor inconvenience. This delay does not cause the product to fail.
What inference optimises for
Inference demands high speed, low cost per request, and consistent reliability. It functions as a continuous process instead of an occasional one.
Training happens when a person starts the process. Inference happens each time a user interacts with the product. In a live application, this can occur thousands of times per minute. The cost of these computations is charged to each individual request.
This clarifies the difference in computing requirements. Training needs the highest possible throughput on the largest available hardware. Inference needs hardware that matches the specific traffic volume. These systems operate continuously and incur high costs if the hardware exceeds what the workload requires.
The cost to train a model does not necessarily indicate the cost to operate it. These two expenses are not closely linked. Teams that monitor only training costs are overlooking the larger share of their total expenditure.
Sources of inference costs
Most teams cannot pinpoint their specific inference costs. They frequently give vague answers or refer to cloud invoices that lack workload detail. These costs arise from three factors: compute resources, memory usage, and traffic patterns.
Compute per request
Each prediction forces the model to perform new calculations, leading to extra compute costs. A small model doing a simple classification task may cost a fraction of a cent per call. A large language model producing a long response costs more because each token needs a separate inference step based on the preceding text. When these individual costs are multiplied by the total number of requests, the expenses become significant once a feature shifts from a demonstration to widespread use.
Memory and hardware requirements
A model must be loaded into memory before it can handle requests. For models larger than a basic size, this demands specialised hardware like a GPU or a dedicated inference chip. This hardware holds the model weights whether or not a request is active. Maintaining this hardware incurs costs even when it is idle. Idle GPU time often causes inference budgets to rise unexpectedly. If you supply enough capacity for peak traffic, you pay for resources that stay unused during quieter periods. If you provide only enough capacity for average traffic, the system will fail during busy periods.
Batching, and its limits
Batching multiple requests together allows the hardware to process them in one pass rather than individually. This method uses the GPU more efficiently and reduces costs for serving setups with high traffic. However, batching increases latency because a request must wait for the batch to fill or for a timeout to occur before processing begins. The reduction in operating costs results in a delay for the user. The decision to use batching depends on the specific requirements of the application.
Latency and throughput pull in different directions
Two metrics describe the performance of an inference system. These metrics represent different concepts. If you confuse them, you may build a system that appears fast on a dashboard but provides a slow experience for the user.
Latency is what one person feels
Latency is the time between a single request going in and its answer coming out. It is what a customer experiences, and it is what gets noticed the moment it slips. A chatbot that takes four seconds to reply feels broken even if it eventually gives a good answer, because people expect conversation to move at conversation speed. Latency is driven by model size, hardware, network hops, and how much work happens before the model even sees the request, and it's measured properly at the tail, the slowest one per cent of requests, rather than the average, because the average hides exactly the failures a customer will hit.
Throughput is what the system handles
Throughput is the total number of requests a system processes during a specific period.
These metrics are what an engineering team monitors on a graph rather than what an individual user experiences. A system can maintain high throughput while providing a poor experience if it achieves that volume by queuing requests into batches. Conversely, a system can have lower throughput but feel responsive to every user if it has enough capacity to process requests without making users wait.
A system designed only for throughput will cause delays for individual users if that allows it to process a high total number of requests per second.
The trade-off has to be a decision, not an accident
Optimising one usually costs you some of the other, and which to favour is a business decision dressed up as an engineering one. A fraud check that runs before a payment clears needs latency measured in tens of milliseconds, whatever that costs in wasted capacity. A nightly model scoring every customer for churn risk needs throughput across millions of rows and does not care if any one row takes half a second, because nobody is staring at a screen waiting for row 40,000. Building the wrong one wastes money in the first case and loses customers in the second.
Batch inference vs real-time inference
Most of the decisions mentioned above depend on a single choice that is usually made early and not changed. You must decide if the prediction needs to occur at the moment it is requested, or if it can occur on a schedule so that it is ready when it is needed.
Batch inference
Batch inference processes predictions for large datasets in a single group. These runs typically occur on a schedule, such as every hour, day, or week, depending on what the specific use case requires. Because no user is waiting for an immediate result while the process runs, the system can be optimised for high throughput and low cost. This allows for large batch sizes and the use of inexpensive hardware that would not be suitable for a live application. Examples of batch processes include calculating churn scores, a demand forecast, or updating product recommendations overnight. The primary disadvantage is latency. Predictions are only as current as the most recent run, so the system will not recognise any data changes that occur between scheduled intervals until the next run begins.
Real-time inference
Real-time inference processes a request as soon as it is received, usually within a service level of several milliseconds. Examples include checking for fraud during a live transaction, ranking search results, generating chatbot responses, or providing recommendations when a user loads a page. These tasks require immediate answers because the prediction is only useful if it occurs during the current interaction. The disadvantage is that the entire serving infrastructure must be scaled to handle peak demand rather than average usage. This requirement results in ongoing costs during periods of low activity.
Picking the wrong one is the actual failure mode
Most teams do not fail because they choose an incorrect model. They fail because they build real-time infrastructure for predictions that could have run in a nightly batch, or they build batch pipelines for cases that required live updates. They often only discover this error when the product team asks why recommendations are a day old. Before beginning any engineering work, you should determine what happens if the result arrives one hour later. If there is no negative impact, you should use a batch process. If the feature requires an instant response to function correctly, you must use a real-time system.
What it means to run inference in production
Generating a single accurate prediction from a model on a local computer represents only a minor portion of the task. The majority of the work involves ensuring the model performs reliably for thousands of different inputs at any time a user requires it.
Serving infrastructure
A model in production operates behind a serving layer. This layer loads the weights, provides an endpoint, and manages concurrent requests. It also scales instances up or down based on traffic volume to maintain performance. This component converts a trained artefact into a functional service for a product. If this layer is built poorly, a model that performed well in a notebook will become unreliable under real traffic. Our overview of model deployment explains this handover process in more detail.
Versioning, and knowing which model answered
Models are frequently retrained, modified, and replaced. Production systems must record which version processed each request. If a failure occurs, you must be able to link a specific prediction to a specific set of weights rather than attributing the error to the model in general. You should not deploy a new version without a method to compare it to the previous version or a way to roll back quickly if performance declines in the live environment. Without these safeguards, a standard update can become a significant incident.
Monitoring what actually happens after launch
A model that performed well during testing may lose accuracy once it handles real traffic. This happens because of changes in incoming data or edge cases that were not included in tests. You cannot see these issues unless you monitor accuracy, latency, and error rates on the live system instead of relying on training reports. Overfitting, which is discussed elsewhere, is a training failure that you can identify before launch. In contrast, inference-time drift only appears after the model has processed real queries for some time. This delay makes the problem difficult to detect.
Where inference fails
Most inference failures do not happen suddenly. The model continues to run, but its performance declines gradually. These changes are often difficult to detect until a specific review is conducted.
Cold starts
When an instance scales down during periods of low activity, it must reload the model into memory to process the next request. This loading process can take several seconds, which results in a slow response time for the first user who connects after the idle period. You can avoid this delay by keeping a warm instance running, though this requires paying for capacity that is not being used. You must choose between the cost of idle resources or the impact of initial latency.
Averages that hide the actual complaint
A system responding in 80 milliseconds on average, with one request in five hundred taking 6 seconds, can seem acceptable when looking at average metrics. However, a user will likely experience the single request taking 6 seconds. Users experience their own individual response times, and not a calculated average. Teams monitoring only average latency are deciding to ignore their slowest performance instances, and these are usually the cause of user complaints.
Where to start
Determine which predictions require instant results and which can run on a schedule before building any infrastructure. This decision affects hardware requirements, total costs, and the on-call responsibilities of the team. Most organisations have fewer genuine real-time needs than they initially assume. Moving even one process to a batch schedule can reduce serving expenses.
Measure your performance before you begin optimising. Record the end-to-end time of an actual request at the tail latency rather than the average. Calculate the cost of one thousand requests based on your intended hardware. Most significant problems in production inference occur because no one took measurements until a customer reported an issue. You can identify the majority of these problems in a single afternoon.
Inference is the part of an AI system that customers interact with, and it requires the same careful development as the model itself. If you are transitioning from a successful demonstration to a system that must perform reliably with actual usage and associated costs, talk to us and we will help you accurately estimate its requirements before it is deployed and changes become difficult.
When the reading turns into a roadmap question, our AI strategy consultants will tell you honestly what the work needs and what it depends on.
Frequently asked questions
What is the meaning of inference in AI?
Inference is the stage where a trained model is put into use. New data is converted into the format used during training, then processed through the model's layers via multiplication and addition until a prediction is generated. This process involves a forward pass only. There is no backward pass and no comparison against a known answer, as no known answer exists to check against. The weights remain identical to how they were set during training, regardless of the input provided.
What is inference in generative AI?
When a generative model like a large language model performs inference, it runs the forward pass multiple times to create output one token at a time. Each new token requires a small inference pass that incorporates all previously generated text. This iterative process is why generating a long response is significantly more expensive than performing a simple classification task. The model remains frozen during inference. It applies the knowledge gained during training rather than learning new information from the specific request.
What is an inference engine in AI?
An inference engine is the serving layer that connects a trained model to a product. It loads the model weights, provides an endpoint, manages concurrent requests, and scales instances based on traffic volume. If this layer is not configured correctly, a model that performed well during development may fail under production traffic. The engine must also include versioning so that specific predictions can be linked to the exact set of weights used to generate them.
What is the cost of inference in AI?
Inference costs result from three factors: the compute required for each prediction, the memory and hardware like GPUs needed to host the model regardless of request volume, and the patterns of the traffic itself. A large language model that produces long responses is significantly more expensive per call than a simple classifier. When these costs are multiplied by total request volume, features that appeared inexpensive during a demonstration can become costly. Batching requests can reduce these expenses, but this approach increases latency.
What is an example of inference in AI?
Real-time inference occurs when a fraud check runs during a transaction, a chatbot generates a reply, or a search ranking appears as a user types a query. In these cases, the system must provide the answer during the interaction. Batch inference occurs when a churn score or demand forecast is updated overnight, because no one is waiting for a specific prediction while the process runs. You should choose between these methods based on whether the result remains useful if it arrives an hour late.