A team we worked with put a chatbot in front of ten customers for a pilot. The transcripts were fine, the answers accurate, and every one of the ten mentioned the same thing unprompted: it felt slow. Not wrong. Slow. Eight seconds between hitting send and the first word appearing, then the rest of the answer sitting there fully typed before it faded into view because the frontend had never been wired for streaming. Nobody defends a product on transcript quality once the person testing it has stopped watching the screen and checked their phone instead.
Latency in AI is the time between a request going in and a usable response coming back, split into time to first token, total response time and perceived latency. It depends on model size, hardware, prefill length and what runs after generation, and setting an honest target starts with measuring each stage separately.
Latency is the time between a request going in and a usable response coming back. That splits into three things people conflate: time to first token, how long someone stares at nothing before anything appears; total response time, how long the whole answer takes to land; and perceived latency, what the wait feels like given how the interface shows progress. A system can be fast on the first measure and still feel slow.
We build data and AI systems at Shipshape Data, and latency is one of the first things that turns a promising pilot into a support ticket. The usual pattern is a team optimising for accuracy in a notebook, shipping it, and discovering that a response which felt instant during testing feels glacial once real users click through it forty times a day. This guide covers where the time in a response actually goes, what model size and hardware do to it, why streaming changes the experience more than the maths, how caching and batching claw seconds back, the trade-off between latency, cost, and quality, and how to set a target that's honest rather than aspirational.
Where the time goes
Ask someone why an AI response takes six seconds and most give the same answer: the model is thinking. Sometimes true, often the smallest part of the story. A request passes through several stages, and each stage can eat time nobody budgeted for.
The trip before the model even starts
Network round trips, authentication, and a load balancer deciding which server handles the request all happen before a single token gets generated. On a good connection this is milliseconds. On a mobile network, or when a request hops through three internal services before reaching the model provider, it adds up to something a user can feel, even though nobody wrote a line of code to cause it.
Prefill and the first token
Before a language model produces a single word of output, it has to process everything sent to it: the prompt, the system instructions, any documents stuffed into the context window. This is called prefill, and it scales with the size of the input. A short question against a short prompt prefills almost instantly. A question sitting behind a system prompt padded with company policy, a couple of retrieved documents, and a chat history grown across twenty turns can push prefill into seconds before generation even begins. This is the most under-diagnosed cause of a slow time to first token, and it's entirely within your control, simply a function of how much you chose to send.
Generation, then everything bolted on
Once prefill finishes, most models generate one token at a time, each depending on the ones before it, which is why response length matters: doubling the answer's length roughly doubles this part of the wait. Then there is whatever the application does before showing the answer: guardrail checks, a second model call to verify or rerank, a tool call to a database, logging that blocks the response instead of running alongside it. We have seen systems where the model answered in under a second and the user still waited four, because a safety check ran serially and nobody had measured it. A call out to retrieval augmented generation before the model sees the question belongs in this category too, and is worth timing on its own rather than folding into "the model was slow".
Measure each stage separately before touching anything. A system slow for one reason is often slow for a different one, and fixing the wrong stage buys you nothing.
Model size and hardware decide the floor
Everything above sits on top of a hard constraint: the model you chose and the hardware it runs on set a floor under how fast a response can be, and no downstream engineering gets you under it.
Bigger is not free
A larger model has more parameters, which usually means better reasoning and broader knowledge, and also means more computation per token. Ten times the parameters doesn't simply mean ten times the wait: the relationship depends on hardware and batching, but it's reliably slower. Reaching for the largest available model because it scores best on a benchmark, when the task at hand is a short classification or a simple extraction, is a common way to buy latency you never needed to spend.
Where it runs
Hardware matters as much as parameter count. GPU memory bandwidth, whether the model is quantised down from full precision, whether it runs from a data centre near your users or one three continents away, whether it competes for capacity with everyone else on the same shared endpoint at the same time of day: all of it moves the number. A quantised model can run meaningfully faster for a small, usually acceptable, quality cost, and for a lot of production cases that's a trade worth taking.
Self-hosted versus API
Calling a hosted model over an API means someone else's infrastructure, queueing, and incident when their region has a bad day. Self-hosting gives you control over hardware and placement, and the price of that control is that you now own the queueing, the scaling, and the incident yourself. A team with unpredictable traffic and no infrastructure appetite is usually better served by an API; a team with steady, high volume and a genuine latency requirement often finds self-hosting's economics worth the operational load. That decision sits alongside the wider question of model deployment, because the latency profile is really a consequence of the deployment choice.
Streaming changes the experience more than the maths
Here is the part that surprises people: two systems can have identical total response time and produce completely different verdicts from users, purely because of whether the answer streams.
Streaming sends tokens to the screen as they are generated instead of holding the whole response until it's complete. Total time to finish is unchanged. What changes is time to first token, the number that actually governs whether a wait feels reasonable. A person watching words appear after two seconds reads the first sentence while the rest generates behind it, barely noticing the six seconds it took to finish. The same person staring at a blank box for six seconds, then getting the whole answer at once, experiences it as a stall followed by a dump of text. The compute cost is identical either way. The perceived cost is not.
Where it earns its keep, and where teams fake it
Streaming matters most in conversational interfaces, where the user is present and watching. It matters far less in a batch job generating overnight reports, and building streaming infrastructure for a pipeline nobody watches live is effort on the wrong problem. We still see products that call a streaming-capable model but buffer the whole response server-side before sending it to the browser, usually because someone needed to run a safety check on the complete text first. That is defensible, but worth knowing as a deliberate trade rather than discovering months later that the "streaming" model in your product has never streamed to a user.
Caching is the cheapest speed you will ever buy
Every millisecond spent regenerating something already computed is a millisecond given away for nothing. Caching is unglamorous and consistently the most effective fix available, because it removes work rather than speeding up the work that remains.
Prompt, response, and semantic caching
Many model providers now let you cache the parts of a prompt that repeat: a long system instruction, a static set of company policies, a document reused across many questions. Instead of prefilling the same tokens again on every call, the cached version is reused and only the new part of the input gets processed. For an application with a heavy fixed system prompt and a light user question on top, this alone can cut time to first token dramatically, and it costs almost nothing to try.
A blunter option is caching whole answers to questions that recur. A support bot fields the same twenty questions from most of its traffic, and serving a cached answer to a question asked a thousand times an hour is close to free speed. The catch is staleness: a cached answer to "what are your opening hours" is fine for months, one to "what is my order status" is wrong the moment it's served twice, so caching has to be scoped to content that's genuinely stable rather than switched on everywhere because it was easy. A more sophisticated version matches on meaning rather than exact text, so "how do I reset my password" and "I forgot my password, help" hit the same cached answer instead of two near-identical model calls, using the same similarity matching behind vector search: the embeddings doing that matching decide whether you are serving the right cached answer or a confidently wrong one to a question that only looked similar.
The fastest tokens are the ones you never generate. Every cache hit is a request the model never had to think about at all.
Batching trades individual speed for total throughput
Batching groups multiple requests together so the hardware processes them as one unit rather than one at a time, which is why a busy production system can serve more traffic per pound than a quiet one, and it directly trades against latency for any single request.
Why providers do it, and what it costs the unlucky request
GPUs are built to do enormous amounts of parallel arithmetic, and running one request through them at a time leaves most of that capacity idle. Batch several together and the same hardware serves all of them for barely more cost than serving one, which is why almost every hosted model API batches under the hood whether you asked for it or not. It is a genuinely good trade for the provider's throughput and a mixed one for the request at the back of the batch, waiting for the whole group to assemble before any of them start. A request landing just as a batch closes waits almost nothing; one landing just after gets held until the next batch fills, a queueing delay that has nothing to do with the model and everything to do with traffic timing. This is invisible in a quiet test environment and very visible at nine in the morning when everyone logs in at once, which is why load testing at realistic concurrency matters more than a laptop with one request at a time.
Where you have control
If you are self-hosting, you set the batching window yourself, and it's a genuine dial rather than a fixed cost: a longer window improves throughput and worsens the tail latency for unlucky requests, a shorter window does the reverse. Getting this right depends on knowing your actual traffic pattern, which is why a proper monitoring setup earns its keep; you can't tune a batching window sensibly against a guess.
Latency, cost, and quality pull against each other
There is no version of this where you get the fastest, cheapest, and best response all at once. Pick any two and the third gives ground, and pretending otherwise is how a project ends up promising something the architecture can't deliver.
The obvious trades
A bigger model is usually better, slower, and pricier. A smaller model is usually faster, cheaper, and worse at anything genuinely hard. Multiple model calls, one to draft and one to check, tend to improve quality and reliably cost both time and money for the privilege. None of this is a surprise once written down. What is surprising is how often a target gets set for one of the three without anyone naming what happened to the other two.
Where the trade gets decided
The honest move is deciding, task by task, which corner matters most. A fraud check running silently in the background can afford to be slow and thorough. A customer-facing chat response can't, and probably shouldn't run the largest model available if a smaller one clears the bar. An internal tool used by three analysts twice a day can absorb a cost nobody would accept at consumer scale. Some try to engineer round the trade-off entirely: smaller model, aggressive caching, batching, quality held steady regardless. Sometimes it genuinely works. More often each change quietly erodes something and nobody notices until a customer does, because each was defensible on its own and the cumulative effect was never measured.
Setting a target that survives contact with real traffic
Most latency targets we see were picked because a round number felt right. "Under two seconds" gets written into a requirements document with no reference to what the task involves or what users tolerate, and six months later somebody is trying to hit a number nobody can justify and nobody wants to relax.
Start from the task, not a number
A quick lookup and a multi-step reasoning task don't deserve the same target, and holding them to one either wastes money keeping the simple task artificially fast or sets the complex task up to fail a bar it was never going to clear. Look at what a person is doing while they wait. A live chat needs a first token inside a couple of seconds or it reads as broken. A report generated once and read later can comfortably take a minute if the content is right. The task tells you the target, not the other way round.
Measure percentiles, and revisit under load
An average response time can look healthy while a meaningful share of users sit through something much worse, because a mean is dragged down by the fast requests and hides the slow ones. Look at the 95th or 99th percentile instead, the number that says how bad the wait gets for the unlucky slice, because that's the experience the people who complain actually had. Then test it under real conditions, not a demo. Batching queues form under real concurrency, hosted APIs slow during their own peak hours, and a cache warmed by test traffic looks nothing like one facing genuinely varied questions. Set the target, then check it again once real traffic has had a few weeks to expose what testing couldn't.
Where to start
Instrument before you optimise. Break real requests down by stage: network, prefill, generation, anything bolted on afterwards, and find out where the seconds go before changing a line of code. Guessing which stage is slow and fixing that one is how teams spend weeks improving a part of the system that was never the bottleneck.
Once you can see the stages, the cheap wins come first: caching for anything genuinely stable, checking whether streaming reaches the user rather than getting buffered somewhere in the middle, and confirming the model in use isn't larger than the task in front of it. Only then does it make sense to talk about hardware, batching windows or a different deployment model, because those are the expensive changes and wasted effort if the cheap ones were never tried.
If your AI product is fast in testing and slow the moment real users touch it, or nobody in the room can say why your latency target is what it is, talk to us. We would rather help you find the actual bottleneck than watch another team throw a bigger GPU at a caching problem.
Frequently asked questions
Where does latency in an AI system go?
A request passes through several stages before an answer appears: network round trips, prefill where the model processes the prompt and any context, then token-by-token generation, and finally anything the application bolts on afterwards such as guardrail checks or a verification pass. Each stage can eat time nobody budgeted for, so measuring them separately matters before changing anything.
How do model size and hardware affect AI latency?
A larger model needs more computation per token, so it's reliably slower even though the relationship isn't a simple multiple of parameter count. Hardware matters just as much: GPU memory bandwidth, whether the model is quantised, and how close it runs to your users all move the number, and a quantised model can run meaningfully faster for a small quality cost.
How does streaming reduce perceived AI latency?
Streaming sends tokens to the screen as they are generated instead of holding the whole response until it finishes, so total response time stays the same but time to first token drops. A person watching words appear after a couple of seconds barely notices the rest of the answer generating behind it, while the same wait with nothing on screen feels like a stall.
How does caching reduce latency in AI systems?
Caching removes work rather than speeding up the work that remains, whether that's caching the repeated part of a prompt so it doesn't get prefilled again, or caching whole answers to questions that recur often. The main risk is staleness, so caching needs scoping to content that's genuinely stable rather than switched on everywhere because it was easy.
How do you set a realistic latency target for an AI system?
A realistic target starts from the task rather than a round number: a live chat needs a first token within a couple of seconds, while a report read later can comfortably take a minute. Measure the 95th or 99th percentile rather than the average, since a mean hides the slow requests that the people who complain experienced.