A team we worked with put a chatbot in front of ten customers for a pilot. The transcripts were fine, the answers accurate, and every one of the ten mentioned the same thing unprompted: it felt slow. Not wrong. Slow. Eight seconds between hitting send and the first word appearing, then the rest of the answer sitting there fully typed before it faded into view because the frontend had never been wired for streaming. Nobody defends a product on transcript quality once the person testing it has stopped watching the screen and checked their phone instead.
Latency is the time between a request going in and a usable response coming back. That splits into three things people conflate: time to first token, how long someone stares at nothing before anything appears; total response time, how long the whole answer takes to land; and perceived latency, what the wait feels like given how the interface shows progress. A system can be fast on the first measure and still feel slow.
We build data and AI systems at Shipshape Data, and latency is one of the first things that turns a promising pilot into a support ticket. The usual pattern is a team optimising for accuracy in a notebook, shipping it, and discovering that a response which felt instant during testing feels glacial once real users click through it forty times a day. This guide covers where the time in a response actually goes, what model size and hardware do to it, why streaming changes the experience more than the maths, how caching and batching claw seconds back, the trade-off between latency, cost and quality, and how to set a target that is honest rather than aspirational.
Where the time actually goes
Ask someone why an AI response takes six seconds and most give the same answer: the model is thinking. Sometimes true, often the smallest part of the story. A request passes through several stages, and each can eat time nobody budgeted for.
The trip before the model even starts
Network round trips, authentication, and a load balancer deciding which server handles the request all happen before a single token gets generated. On a good connection this is milliseconds. On a mobile network, or when a request hops through three internal services before reaching the model provider, it adds up to something a user can feel, even though nobody wrote a line of code to cause it.
Prefill and the first token
Before a language model produces a single word of output, it has to process everything sent to it: the prompt, the system instructions, any documents stuffed into the context window. This is called prefill, and it scales with the size of the input. A short question against a short prompt prefills almost instantly. A question sitting behind a system prompt padded with company policy, a couple of retrieved documents, and a chat history grown across twenty turns can push prefill into seconds before generation even begins. This is the most under-diagnosed cause of a slow time to first token, and it is entirely within your control, simply a function of how much you chose to send.
Generation, then everything bolted on
Once prefill finishes, most models generate one token at a time, each depending on the ones before it, which is why response length matters: doubling the answer's length roughly doubles this part of the wait. Then there is whatever the application does before showing the answer: guardrail checks, a second model call to verify or rerank, a tool call to a database, logging that blocks the response instead of running alongside it. We have seen systems where the model answered in under a second and the user still waited four, because a safety check ran serially and nobody had measured it. A call out to retrieval augmented generation before the model sees the question belongs in this category too, and is worth timing on its own rather than folding into "the model was slow".
Measure each stage separately before touching anything. A system slow for one reason is often actually slow for a different one, and fixing the wrong stage buys you nothing.
Model size and hardware decide the floor
Everything above sits on top of a hard constraint: the model you chose and the hardware it runs on set a floor under how fast a response can be, and no downstream engineering gets you under it.
Bigger is not free
A larger model has more parameters, which usually means better reasoning and broader knowledge, and also means more computation per token. Ten times the parameters does not simply mean ten times the wait, the relationship depends on hardware and batching, but it is reliably slower. Reaching for the largest available model because it scores best on a benchmark, when the task at hand is a short classification or a simple extraction, is a common way to buy latency you never needed to spend.
Where it actually runs
Hardware matters as much as parameter count. GPU memory bandwidth, whether the model is quantised down from full precision, whether it runs from a data centre near your users or one three continents away, whether it competes for capacity with everyone else on the same shared endpoint at the same time of day: all of it moves the number. A quantised model can run meaningfully faster for a small, usually acceptable, quality cost, and for a lot of production cases that is a trade worth taking.
Self-hosted versus API
Calling a hosted model over an API means someone else's infrastructure, queueing and incident when their region has a bad day. Self-hosting gives you control over hardware and placement, and the price of that control is that you now own the queueing, the scaling and the incident yourself. A team with unpredictable traffic and no infrastructure appetite is usually better served by an API; a team with steady, high volume and a genuine latency requirement often finds self-hosting's economics worth the operational load. That decision sits alongside the wider question of model deployment, because the latency profile is really a consequence of the deployment choice.
Streaming changes the experience more than the maths
Here is the part that surprises people: two systems can have identical total response time and produce completely different verdicts from users, purely because of whether the answer streams.
Streaming sends tokens to the screen as they are generated instead of holding the whole response until it is complete. Total time to finish is unchanged. What changes is time to first token, the number that actually governs whether a wait feels reasonable. A person watching words appear after two seconds reads the first sentence while the rest generates behind it, barely noticing the six seconds it took to finish. The same person staring at a blank box for six seconds, then getting the whole answer at once, experiences it as a stall followed by a dump of text. The compute cost is identical either way. The perceived cost is not.
Where it earns its keep, and where teams fake it
Streaming matters most in conversational interfaces, where the user is present and watching. It matters far less in a batch job generating overnight reports, and building streaming infrastructure for a pipeline nobody watches live is effort on the wrong problem. We still see products that call a streaming-capable model but buffer the whole response server-side before sending it to the browser, usually because someone needed to run a safety check on the complete text first. That is defensible, but worth knowing as a deliberate trade rather than discovering months later that the "streaming" model in your product has never actually streamed to a user.
Caching is the cheapest speed you will ever buy
Every millisecond spent regenerating something already computed is a millisecond given away for nothing. Caching is unglamorous and consistently the most effective fix available, because it removes work rather than speeding up the work that remains.
Prompt, response and semantic caching
Many model providers now let you cache the parts of a prompt that repeat: a long system instruction, a static set of company policies, a document reused across many questions. Instead of prefilling the same tokens again on every call, the cached version is reused and only the new part of the input gets processed. For an application with a heavy fixed system prompt and a light user question on top, this alone can cut time to first token dramatically, and it costs almost nothing to try.
A blunter option is caching whole answers to questions that recur. A support bot fields the same twenty questions from most of its traffic, and serving a cached answer to a question asked a thousand times an hour is close to free speed. The catch is staleness: a cached answer to "what are your opening hours" is fine for months, one to "what is my order status" is wrong the moment it is served twice, so caching has to be scoped to content that is genuinely stable rather than switched on everywhere because it was easy. A more sophisticated version matches on meaning rather than exact text, so "how do I reset my password" and "I forgot my password, help" hit the same cached answer instead of two near-identical model calls, using the same similarity matching behind vector search: the embeddings doing that matching decide whether you are serving the right cached answer or a confidently wrong one to a question that only looked similar.
The fastest tokens are the ones you never generate. Every cache hit is a request the model never had to think about at all.
Batching trades individual speed for total throughput
Batching groups multiple requests together so the hardware processes them as one unit rather than one at a time, which is why a busy production system can serve more traffic per pound than a quiet one, and it directly trades against latency for any single request.
Why providers do it, and what it costs the unlucky request
GPUs are built to do enormous amounts of parallel arithmetic, and running one request through them at a time leaves most of that capacity idle. Batch several together and the same hardware serves all of them for barely more cost than serving one, which is why almost every hosted model API batches under the hood whether you asked for it or not. It is a genuinely good trade for the provider's throughput and a mixed one for the request at the back of the batch, waiting for the whole group to assemble before any of them start. A request landing just as a batch closes waits almost nothing; one landing just after gets held until the next batch fills, a queueing delay that has nothing to do with the model and everything to do with traffic timing. This is invisible in a quiet test environment and very visible at nine in the morning when everyone logs in at once, which is why load testing at realistic concurrency matters more than a laptop with one request at a time.
Where you have control
If you are self-hosting, you set the batching window yourself, and it is a genuine dial rather than a fixed cost: a longer window improves throughput and worsens the tail latency for unlucky requests, a shorter window does the reverse. Getting this right depends on knowing your actual traffic pattern, which is why a proper monitoring setup earns its keep; you cannot tune a batching window sensibly against a guess.
Latency, cost and quality pull against each other
There is no version of this where you get the fastest, cheapest and best response all at once. Pick any two and the third gives ground, and pretending otherwise is how a project ends up promising something the architecture cannot deliver.
The obvious trades
A bigger model is usually better, slower and pricier. A smaller model is usually faster, cheaper and worse at anything genuinely hard. Multiple model calls, one to draft and one to check, tend to improve quality and reliably cost both time and money for the privilege. None of this is a surprise once written down. What is surprising is how often a target gets set for one of the three without anyone naming what happened to the other two.
Where the trade actually gets decided
The honest move is deciding, task by task, which corner matters most. A fraud check running silently in the background can afford to be slow and thorough. A customer-facing chat response cannot, and probably should not run the largest model available if a smaller one clears the bar. An internal tool used by three analysts twice a day can absorb a cost nobody would accept at consumer scale. Some try to engineer round the trade-off entirely: smaller model, aggressive caching, batching, quality held steady regardless. Sometimes it genuinely works. More often each change quietly erodes something and nobody notices until a customer does, because each was defensible on its own and the cumulative effect was never measured.
Setting a target that survives contact with real traffic
Most latency targets we see were picked because a round number felt right. "Under two seconds" gets written into a requirements document with no reference to what the task involves or what users actually tolerate, and six months later somebody is trying to hit a number nobody can justify and nobody wants to relax.
Start from the task, not a number
A quick lookup and a multi-step reasoning task do not deserve the same target, and holding them to one either wastes money keeping the simple task artificially fast or sets the complex task up to fail a bar it was never going to clear. Look at what a person is doing while they wait. A live chat needs a first token inside a couple of seconds or it reads as broken. A report generated once and read later can comfortably take a minute if the content is right. The task tells you the target, not the other way round.
Measure percentiles, and revisit under load
An average response time can look healthy while a meaningful share of users sit through something much worse, because a mean is dragged down by the fast requests and hides the slow ones. Look at the 95th or 99th percentile instead, the number that says how bad the wait gets for the unlucky slice, because that is the experience the people who complain actually had. Then test it under real conditions, not a demo. Batching queues form under real concurrency, hosted APIs slow during their own peak hours, and a cache warmed by test traffic looks nothing like one facing genuinely varied questions. Set the target, then check it again once real traffic has had a few weeks to expose what testing could not.
Where to start
Instrument before you optimise. Break real requests down by stage: network, prefill, generation, anything bolted on afterwards, and find out where the seconds go before changing a line of code. Guessing which stage is slow and fixing that one is how teams spend weeks improving a part of the system that was never the bottleneck.
Once you can see the stages, the cheap wins come first: caching for anything genuinely stable, checking whether streaming actually reaches the user rather than getting buffered somewhere in the middle, and confirming the model in use is not larger than the task in front of it. Only then does it make sense to talk about hardware, batching windows or a different deployment model, because those are the expensive changes and wasted effort if the cheap ones were never tried.
If your AI product is fast in testing and slow the moment real users touch it, or nobody in the room can say why your latency target is what it is, talk to us. We would rather help you find the actual bottleneck than watch another team throw a bigger GPU at a caching problem.