A client fed an entire year of support tickets into a single prompt and asked for the three biggest product complaints. The model answered fluently, in tidy prose, with a confident little ranking. It was also wrong. The tickets that mattered most sat in the middle of the file, not the start or the end, and that is roughly where a model's attention tends to go thin. Nobody had told it the file was too big to hold onto properly. It never mentioned that either. It just answered.
A context window is the amount of text a model can hold in view at once while it works: the system prompt, any documents pasted in, the back-and-forth of the conversation so far, and the reply it is currently writing, all counted against the same budget. Go past that budget and something has to give. What gives depends on the system in front of you, and it is rarely obvious to the person typing that anything gave at all.
We build data and AI systems at Shipshape Data, and context windows are one of those things that looks like a footnote until a client's proof of concept quietly stops working the moment the document set grows past a few dozen pages. This guide covers what actually gets counted as a token, what happens when you go over the limit, why the cost curve is steeper than most people assume, why a bigger window rarely fixes the underlying problem, and the retrieval and chunking work that gets a team past the limit properly rather than around it.
What actually fills a context window
Context windows are measured in tokens, and a token is not a word. It is a chunk of text the model's tokeniser has decided to treat as one unit, which might be a whole common word, half a word, a punctuation mark, or a single character if the text is unusual enough. As a rough working figure, English text runs at something like three to four characters per token, though that shifts with the language, the formatting, and how much of the text is code or numbers rather than prose.
Tokens are not words
This matters because people estimate context budgets in words and then get surprised. A ten-thousand-word report is not ten thousand tokens. It is closer to thirteen or fourteen thousand once formatting, headers and the odd stray symbol are counted. Tables are worse: a spreadsheet pasted as text can burn through tokens far faster than the same information written as prose, because every cell boundary and repeated label costs something. Code is its own case again, since indentation, brackets and repeated variable names all tokenise differently to ordinary sentences, which is why a modest script can eat far more of the budget than its line count suggests.
Non-text input complicates the estimate further. An image gets converted into a fixed block of tokens depending on its resolution, and a PDF handed to a model is often processed page by page as images plus extracted text, so a scanned document can cost noticeably more than the same words typed cleanly. None of this is visible from the file on your desktop. It only shows up once you are inside the system watching the counter climb.
Everything shares one budget
The number quoted for a model, a hundred thousand tokens, a million tokens, whatever it happens to be, covers a lot more than your question. It covers the system prompt the application sends behind the scenes, every document you have attached, the entire conversation history up to that point, and the output the model is about to generate. In a long-running chat, that history keeps compounding. Turn one might use a few hundred tokens. By turn forty, if nothing has been trimmed, you could be resending the whole conversation every single time just to ask one more question.
What happens when you exceed it
There are two ways this goes wrong, and only one of them tells you it happened.
The hard stop
Call an API with more tokens than the model accepts and you get an error. The request is rejected outright, nothing gets generated, and the fix is obvious: cut something. This is the easy failure mode, because it is loud. Nobody ships a product on top of it without noticing.
The silent version is worse
Chat interfaces rarely let a conversation hit a hard wall, because that would be a poor experience. Instead they manage the budget for you, and the usual method is to quietly drop or summarise the earliest turns once the window fills up. That instruction you gave at the start of the conversation, the formatting preference, the constraint about not mentioning a competitor, can simply fall out of scope while the conversation keeps going as if nothing changed. Nobody sees an error. The tone just drifts, or an instruction stops being followed, and it takes a while to work out why.
Even within a window that technically fits, position matters. Content placed early or late in a long prompt tends to get weighted more heavily than content buried in the middle, an effect researchers studying long-context retrieval have documented and generally call "lost in the middle". Practically, it means a fact your model needs is more likely to be missed if it sits on page forty of a ninety-page document than if it sits on page one or page ninety.
We have seen this play out with a client's internal search tool, built to answer questions against an onboarding handbook. Short questions against a short handbook worked fine in testing. Once the handbook grew to cover three departments and the tool started stuffing the whole thing into every query, answers about a policy buried mid-document started coming back thin or slightly off, while questions about the first and last sections stayed sharp. The window technically still fit. The model's attention did not spread evenly across it.
Why more tokens costs more than you'd think
Providers charge per token, input and output separately, and that alone should make the arithmetic obvious: paste in twice as much text and you pay roughly twice as much just to have the model read it, before it has written a word back.
The turn you don't see repeating
The part people miss is that a conversational system typically resends the full history with every single turn, because the model has no memory of its own between calls. Ask a tenth question in a long chat and you are not paying for one question. You are paying to re-read the previous nine, plus their answers, plus the new one. A conversation that started cheap can get expensive purely through its own length, with no new information being added at all.
The underlying computation gets heavier too, on top of the raw token count. The attention mechanism inside a transformer compares every token to every other token, so cost and latency scale worse than linearly as the sequence grows. Doubling the input is not simply double the work. Some providers offer prompt caching to soften this, charging less to reprocess a prefix the model has already seen, which helps with a stable system prompt or a document you query repeatedly, but it does not change the underlying shape of the problem, only the price of living with it.
Why a bigger window is not automatically better
The obvious response to all of this is to reach for a model with a larger window and stop worrying. That works for some problems and quietly makes others worse.
A bigger window means more room for irrelevant text to sit next to the answer you actually need, and irrelevant text is not free just because it fits. It has to be read, weighed, and mostly discarded, and the discarding is where things go wrong. Give a model your entire product manual when the question only concerns one setting, and you have handed it a much larger haystack to search for the same needle, with more chances for something unrelated to get pulled into the answer because it happened to sound plausible next to the real content.
A bigger context window does not make a model more careful. It gives a careless prompt more room to hide in.
There is also the simple fact that a huge window invites a huge prompt, and a huge prompt is usually a sign nobody has done the work of deciding what actually matters. Pasting in everything is easier than curating, right up until the answer comes back vague, or confidently wrong, or several pounds more expensive than it needed to be.
Retrieval: giving the model only what it needs
The alternative to stuffing everything into the prompt is to fetch only the relevant slice of it at query time, and answer from that. This is retrieval-augmented generation, usually shortened to RAG, and it is the standard answer to "our documents don't fit in the window" once the document set is anything more than a handful of files.
The mechanics are not exotic. Documents get broken into pieces, each piece is converted into an embedding, a numerical representation of its meaning, and those embeddings are stored in a vector database. When a question comes in, it gets embedded the same way, the database finds the pieces whose meaning sits closest to it, and only those pieces, usually a handful of paragraphs rather than a whole library, get passed to the model alongside the question. The model never sees the full document set. It sees the parts that were judged relevant, which is a much smaller and much more targeted context to reason over.
Done well, this keeps cost roughly flat as the knowledge base grows, because the amount fed to the model per query does not change even if the underlying library doubles. Done badly, it just moves the failure somewhere less visible: if retrieval pulls back the wrong chunks, the model will answer fluently from the wrong evidence, and there is no error message for that. It looks exactly like a correct answer until someone checks.
Pure semantic search on embeddings is not the whole answer either, and treating it as one is a common early mistake. Meaning-based matching is excellent at finding a paragraph that discusses the same idea in different words, and genuinely weak at exact matching: a product code, an order number, a specific clause reference. A hybrid approach that combines embedding search with straightforward keyword matching, then blends or reranks the results, tends to outperform either method run alone, particularly on the kind of precise lookup that business users actually ask for most often.
Chunking: the part that quietly decides whether retrieval works
Retrieval is only as good as the pieces it retrieves from, and deciding how to cut a document into pieces, chunking, is where most of the actual difficulty in RAG sits. It gets treated as a plumbing detail. It behaves more like the decision that determines whether the whole system works.
Too small loses the point, too big loses the precision
Chunk too finely, splitting by sentence or by a fixed small number of tokens, and you get pieces that are individually meaningless out of context. A clause that only makes sense next to the paragraph before it will retrieve on its own and confuse the model that receives it. Chunk too coarsely, whole sections or entire documents, and retrieval brings back a wall of text again, most of it irrelevant to the actual question, which is the exact problem retrieval was meant to solve.
The usual middle ground is chunking by a few hundred tokens with a modest overlap between consecutive chunks, so a fact that lands near a boundary is not orphaned in one piece and missing from the other. Better still is chunking along natural structure, headings, sections, paragraphs, rather than a blind token count, because a chunk that respects the document's own logic tends to retrieve as a coherent, self-contained idea rather than a fragment.
Metadata does more work than people expect
A chunk stripped of its source loses something important: which document it came from, which section, how recent it is. Attach that metadata at chunking time and you can filter retrieval by recency, or trace an answer back to the exact paragraph it came from, which matters enormously the first time someone asks why the model said what it said. Skip it, and every chunk is an anonymous floating paragraph, indistinguishable from any other, with no way to tell a policy from 2019 apart from its replacement.
This is also where document rot quietly resurfaces as a data problem rather than a modelling one. If the source library holds three versions of the same policy because nobody archived the old ones, chunking and embedding all three faithfully preserves that mess. The model will retrieve whichever version scores closest to the question, with no particular bias towards the current one, and hand back an answer that reads as authoritative regardless of which year it came from. Cleaning up the source library before it gets chunked saves far more grief than any amount of clever retrieval tuning afterwards.
Where to start
Work out what your actual use case needs before choosing a model on window size alone. If the task is genuinely about reasoning across one bounded document, a long-context model used carefully is the right tool, and there is no need to build a retrieval system for a single fifty-page contract. If the task is answering questions against a knowledge base that grows over time, retrieval is the right shape from day one, however small that base feels today.
Measure token counts rather than guessing them, on real documents rather than clean samples, because the messy ones are where the estimate breaks. Then look honestly at your chunking. Pull ten answers from whatever retrieval setup you have and check what actually got fetched for each one; a surprising number of poor answers turn out to be poor retrieval wearing a confident model's voice, not a model problem at all.
None of this needs solving in one sitting, and it rarely gets solved well by adding more tokens to the budget and hoping. If you are weighing up whether your AI project needs a retrieval layer, or you are already running one and not sure why the answers still miss, talk to us.