We collaborated with a team that evaluated a chatbot using ten customers in a pilot programme. The transcripts indicated that the answers were accurate. However, all ten participants spontaneously noted that the system felt slow. An eight-second lag occurred between sending a message and the appearance of the first word. The complete answer then loaded fully before fading into view, since the frontend was not built for streaming. Users disregard transcript quality when the delay makes them look away from the screen and check their phones.
AI latency refers to the time elapsed from a user request to the provision of a usable response. The measurement covers time to first token, total response time, and perceived latency. Model size, hardware specifications, prefill length, and post-generation processes influence these metrics. To establish precise performance targets, each of these stages must be measured individually.
Latency measures the time from when a user makes a request to when a usable response is received. This interval includes three separate factors that are frequently confused. The first factor is time to first token, which represents how long a user waits before any content appears. The second factor is total response time, which denotes the time needed for the complete answer to be delivered. The third factor is perceived latency, which reflects the user's subjective sense of the wait as shaped by the interface's progress display. A system can score well on the first metric yet still feel slow to the user.
At Shipshape Data we build data and AI systems, and latency frequently becomes the first problem that turns a promising pilot project into a source of support requests. Usually a team concentrates on accuracy while creating a system in a notebook environment, then deploys it, only to discover that a response time that appeared fast in testing becomes unacceptably slow when many users use it in the real world. This guide describes where response time is consumed, how model size and hardware influence it, why streaming affects user experience beyond its mathematical impact, how caching and batching can lower response times, the connection between latency, cost, and quality, and how to set a realistic and justifiable performance target.
Where the time goes
When you enquire about why an AI response takes six seconds, most people answer that the model is thinking. Although this can be true at times, it is frequently only a minor factor. A request goes through several stages, and each stage may require more time than expected.
The trip before the model even starts
Network round trips, authentication, and load balancing all happen before token generation starts. On a stable connection, these processes take a few milliseconds. However, the delay becomes noticeable to users on mobile networks or when a request traverses multiple internal services before reaching the model provider. This latency appears even though it is not explicitly programmed into the application code.
Prefill and the first token
A language model must process all supplied data before it generates any output. This includes the prompt, system instructions, and any documents within the context window. This stage is called prefill. The time required to complete scales with the amount of input. A short question paired with a brief prompt processes quickly. However, a question that follows long system instructions, retrieved documents, and an extensive chat history can cause a delay of several seconds before generation begins. This is the most common cause of a slow time to first token. You can control this delay by limiting the amount of information sent to the model.
Generation, then everything bolted on
After the prefill step completes, most models produce tokens sequentially, each token relying on the ones before it. Therefore response length is important; doubling the answer length roughly doubles this portion of the waiting time. The application also adds time before showing the answer, such as guardrail checks, a second model call for verification or reranking, a database tool call, and logging that might have occurred concurrently with response generation. We have observed systems where the model responded in under a second, yet the user waited four seconds because a safety check ran sequentially and its duration was not measured. A call to retrieval augmented generation before the model processes the question is also part of this category and should be timed separately from the period when "the model was slow".
Assess each stage separately before implementing any changes. A system that is slow due to one cause is frequently slow due to another cause as well. Correcting the wrong stage will not enhance performance.
Model size and hardware decide the floor
The chosen model and hardware set a fixed ceiling on response speed. No engineering effort made after these selections can lower the response time beneath this baseline.
Bigger is not free
Models that have more parameters usually show better reasoning and a wider knowledge base. These models also need more computation per token they generate. Increasing the parameter count by ten does not automatically cause a tenfold rise in wait time, because actual speed depends on hardware and batching. Nevertheless, larger models are consistently slower. Users frequently select the largest model due to high benchmark scores on tasks such as short classification or simple extraction. This approach leads to unnecessary latency.
Where it runs
Hardware is equally important as the number of parameters. Performance is influenced by several factors, such as GPU memory bandwidth and whether the model employs quantisation rather than full precision. The physical distance from the data centre to users also alters the results, as does the traffic volume on a shared endpoint at a given time of day. A quantised model typically runs faster while sacrificing a small amount of quality. In many production settings, this speed gain is a practical trade-off.
Self-hosted versus API
Using a hosted model through an API depends on external infrastructure, queuing systems, and potential service outages in certain regions. Self-hosting gives control over hardware and location, yet it requires you to handle your own queuing, scaling, and technical incidents. Teams that face unpredictable traffic and have little interest in managing infrastructure usually prefer an API. Teams with steady, high volume and strict latency needs often conclude that the cost savings of self-hosting justify the operational effort. This decision is part of the broader process of model deployment, because the deployment method determines the resulting latency profile.
Streaming changes the experience more than the maths
Two systems may share the same total response time yet receive different user feedback. This occurs due to the way the systems deliver answers. Users view a system differently when it streams the response rather than delivering it all at once.
Streaming shows tokens as they are produced instead of waiting for the whole response to complete. The overall completion time stays unchanged. Nevertheless, streaming shortens the time to the first token, which is the measure that shapes a user's perception of the wait. A user who observes words start appearing after two seconds can read the initial sentence while the rest of the text is generated. They probably will not notice that the entire process lasted six seconds. If the same user waits six seconds for the whole answer to appear at once, they will view the delay as a system stall. The computational cost is identical for both approaches, yet the user experience differs.
Where it earns its keep, and where teams fake it
Streaming is most beneficial in conversational interfaces since the user observes the response as it unfolds. It matters less for batch jobs that produce reports overnight. Creating streaming infrastructure for processes not displayed live is an inefficient use of resources. Some products employ models that can stream yet wait for the complete response to finish on the server before delivering it to the browser. This frequently occurs because the system needs to conduct a safety check on the whole text. This method is acceptable when it is a deliberate choice. You ought to verify how your system processes these calls so you do not later discover that your product never actually streamed data to the user.
Caching is the cheapest speed you will ever buy
Regenerating data that has already been computed wastes time.
Caching is a straightforward yet highly effective optimisation. It enhances performance by removing tasks rather than trying to accelerate existing processes.
Prompt, response, and semantic caching
Many model providers now allow caching of repeating parts of a prompt: a lengthy system instruction, a static collection of company policies, or a document reused across many questions. Rather than prefilling the same tokens on each call, the cached version is reused and only the new portion of the input is processed. For an application that has a large fixed system prompt plus a brief user question, this alone can dramatically reduce time to first token, and it costs virtually nothing to attempt.
You can cache complete answers to frequently asked questions. Support bots frequently receive the same twenty questions from most users. Supplying a cached answer to a question asked a thousand times per hour boosts speed at virtually no cost; the primary issue is staleness. A cached answer to "what are your opening hours" stays accurate for months. However, an answer to "what is my order status" becomes incorrect if reused. You should restrict caching to stable content instead of applying it to every query. A more advanced technique matches the meaning of a question rather than the exact text. This enables "how do I reset my password" and "I forgot my password, help" to trigger the same cached response. This avoids two nearly identical model calls by employing the same similarity matching used in vector search. The embeddings used for this matching decide whether you are delivering the correct cached answer or an incorrect one for a question that only seemed similar.
The quickest tokens are the ones that are not generated. Each cache hit signifies a request that the model does not have to process.
Batching trades individual speed for total throughput
Batching merges multiple requests into a single unit for hardware processing. This method enables a high-volume production system to manage more traffic per resource than a low-volume system. Applying this technique raises the hardware's processing capacity, yet it also raises the latency for each individual request.
Why providers do it, and what it costs the unlucky request
GPUs are built to execute large amounts of parallel arithmetic. Processing one request at a time leaves most of that capacity idle. If several requests are batched, the hardware handles all of them at a cost almost equal to handling a single request. Therefore most hosted model APIs enable batching by default. The method raises throughput for the provider, yet it affects individual requests in different ways. A request must wait until the whole group is assembled before processing can start. A request arriving right as a batch closes sees virtually no delay. A request arriving just after a batch closes is held until the next batch becomes full. This queuing delay results from traffic timing, not from the model itself. These delays are not visible in quiet test environments, yet they become clear during peak usage when many users log in at the same time. Consequently, load testing with realistic concurrency is more valuable than testing a single request on a laptop.
Where you have control
When you host the service yourself, you control the batching window. This setting acts as a variable adjustment instead of a fixed cost. A longer window raises throughput but also raises tail latency for some requests. A shorter window lowers latency but reduces throughput. To select the proper setting, you need to understand your specific traffic patterns. A functional monitoring system is required for this task, because you cannot optimise the batching window using estimates.
Latency, cost, and quality compete with each other
You cannot get the fastest, cheapest, and highest-quality response at the same time. You must pick two of these priorities, causing the third to be compromised. Ignoring this trade-off creates project goals that the technical architecture cannot support.
The obvious trades
Large models usually deliver better results but run more slowly and cost more. Small models are faster and cheaper, although they perform poorly on complex tasks. Employing multiple model calls to draft and review content generally raises quality, but this method needs more time and higher expenses; these trade-offs are predictable. However, people often set goals for one factor without considering the impact on the other two.
Where the trade gets decided
For each task you should determine which specific requirement is most important. A fraud check that runs in the background may be slow and thorough. A chat response for customers must not be slow, and it is generally preferable to use a smaller model if it satisfies the quality standard. An internal tool used by a few analysts twice a day can incur high costs that would be unacceptable for a consumer product. Some people try to avoid these trade-offs by employing smaller models, aggressive caching, and batching while aiming to keep quality steady. This approach works occasionally. More often, these changes gradually reduce quality in ways that go unnoticed until a customer complains. This occurs because each individual change appears reasonable, but the total impact is never measured.
Setting a target that survives contact with real traffic
Most latency targets are chosen because a round number appears appropriate. People write "Under two seconds" into requirements documents without considering the specific task or user tolerance. Six months later, staff must try to meet a target that has no justification and that no one is willing to change.
Start from the task, not a number
Simple lookups and complex reasoning tasks need different performance targets. Applying the same standard to both results in unnecessary costs for simple tasks or unrealistic expectations for complex ones. You should assess what the user is doing during the wait time. A live chat interface must deliver a response within a few seconds to appear functional. A report generated for later reading can take a minute to complete if the quality is high. The nature of the task sets the appropriate speed target.
Measure percentiles, and revisit under load
Average response times may look acceptable while many users still face significant delays. This occurs because fast requests pull the mean down and hide the slower ones. Instead, you should track the 95th or 99th percentile. These metrics display the maximum wait time for the slowest group of users, reflecting the real experience of those who report problems. You also need to test performance under realistic conditions instead of using a demo environment. Real concurrency creates queues, and third-party APIs frequently slow down during their own peak usage periods. Furthermore, a cache populated by test traffic behaves differently than one that handles a diverse range of real queries. Set your performance targets and then re-evaluate them after several weeks of live traffic have uncovered issues that testing did not reveal.
Where to start
Measure your performance before starting optimisation. Split actual requests into distinct stages, for example network, prefill, generation, and any later processes. Determine precisely where time is spent before changing any code. If you guess the slow stage and try to fix it without data, you could waste weeks improving a part of the system that does not constrain overall speed.
Once you have identified the different stages, tackle the simple improvements first. You should cache stable data, make sure streaming reaches the user without buffering by intermediate systems, and confirm that the model size fits the task. Only after these steps should you consider hardware upgrades, batching windows, or new deployment models. These larger changes take more effort and are unnecessary if the simpler optimisations resolve the problem.
If your AI product works well in testing yet decelerates once real users access it, or if your team cannot clarify the reasoning behind your current latency targets, talk to us. We aim to help you pinpoint the exact cause of a bottleneck rather than watching a team resort to pricier hardware to fix a problem that stems from the cache.
Beyond the reading stage and ready to build? Our AI development team delivers systems like this using governed data, scoped for production from day one.
Frequently asked questions
Where does latency in an AI system go?
A request passes through several stages before a response is generated. These stages comprise network round trips, the prefill stage in which the model handles the prompt and context, token-by-token generation, and final application steps such as guardrail checks or verification. Any stage may take longer than anticipated. Measure each stage individually before making any changes.
How do model size and hardware affect AI latency?
Larger models need more computation per token. Consequently they run slower, but the slowdown is not directly proportional to the parameter count. Hardware influences performance as well. GPU memory bandwidth, the application of quantisation, and the physical distance between the model and the users all modify the speed. A quantised model can operate considerably faster while sacrificing only a small amount of quality.
How does streaming reduce perceived AI latency?
Streaming shows tokens as the system produces them. This does not alter the overall response time, but it shortens the interval before the first token is visible. A user who observes words appearing within a few seconds is unlikely to be disturbed by the rest of the generation process. Yet, waiting the same duration for a full response with an empty screen feels like a delay.
How does caching reduce latency in AI systems?
Caching removes repetitive work. It may keep repeated sections of a prompt to prevent redundant prefilling, or it may retain full answers to common questions. The main risk of this approach is outdated information. Consequently, caching ought to be used only for content that stays consistent over time.
How do you set a realistic latency target for an AI system?
Define a performance target according to the particular task rather than choosing a round number. A live chat needs a response within a few seconds, whereas a report meant for later reading may take up to a minute. Measure the 95th or 99th percentile. Relying on the mean average conceals the slow requests that cause user complaints.