A client placed one year of support tickets into a single prompt to find the three most common product complaints. The model returned a clear, ranked list in reply. Yet the information was wrong. The most significant tickets were situated in the middle of the file, which is the area where models often lose focus. The model was not told that the file exceeded its effective processing capacity. It gave an answer without reporting any limitations.
A context window is the total number of tokens a language model can process at one time. This limit covers the system prompt, uploaded documents, the conversation history, and the response the model is currently generating. If the input goes beyond this limit, the system will either produce an error or continue while discarding the earliest parts of the conversation from its memory.
A context window represents the total amount of text that a model can handle in a single pass. This limit covers the system prompt, any attached documents, the history of the current conversation, and the response the model is generating. When the text goes beyond this limit, the model must leave out some information. The exact approach for dealing with this overflow varies by system, and users often do not notice that information has been removed.
Shipshape Data creates data and AI systems. Context windows often appear to be a minor detail until a client's proof of concept fails once the document set exceeds a few dozen pages. This guide explains how tokens are counted and what happens when the limit is exceeded. It also describes why costs rise faster than expected and why a larger window usually does not solve the main issue. Finally, it covers the retrieval and chunking methods needed to manage these limits effectively.
What actually fills a context window
Context windows are quantified in tokens instead of words. A token is a piece of text that the model's tokeniser handles as one unit. That unit may be a common word, a fragment of a word, a punctuation mark, or a single character when the text is uncommon. In English text, one token typically corresponds to three to four characters. The ratio varies according to the language, the formatting, and the quantity of code or numbers present.
Tokens are not words
People frequently underestimate context budgets when they base calculations on word counts. A report with ten thousand words does not equal ten thousand tokens. When formatting, headers, and symbols are added, the total typically rises to around thirteen or fourteen thousand tokens. Tables take up even more tokens. Pasting a spreadsheet as text consumes tokens far more quickly than presenting the same data in prose, since cell borders and repeated labels add extra tokens. Code behaves differently as well. Indentation, brackets, and repeated variable names are tokenised in ways ordinary sentences are not. Thus, a small script can consume considerably more of the budget than its line count would suggest.
Non-text input complicates the estimation. An image is transformed into a defined number of tokens according to its resolution. When a PDF is supplied, the model typically processes it page by page as both images and extracted text. As a result, a scanned document may be more costly than a text file containing the same content. These costs cannot be determined by simply looking at the file on your computer. The actual cost becomes visible only when the system starts counting tokens during processing.
Everything shares one budget
The token limit given for a model, whether it is one hundred thousand or one million tokens, covers several elements beyond your specific question. It includes the system prompt sent by the application, any attached documents, the complete conversation history, and the model's response. In an extended chat, this history builds up. The first turn may use only a few hundred tokens. By the fortieth turn, if the history is not trimmed, you could be sending the whole conversation again each time you pose a new question.
What happens when you exceed it
This process can fail in two manners. Only one of those failures sends a notification that it happened.
The hard stop
Calling an API with more tokens than the model permits results in an error. The system immediately rejects the request and provides no response. This is an easy failure to address since the error is obvious. You need to shorten the input to resolve the issue.
No one releases a product with this without being aware of it.
The silent version is worse
Chat interfaces seldom permit a conversation to end abruptly because that would produce a poor user experience. Instead, they handle the context window by deleting or summarising the earliest dialogue portions when the limit is reached. An instruction given at the start of the conversation-like a formatting preference or a competitor constraint-may drop out of the active scope as the interaction proceeds. No error message is shown. The response tone can shift or the system may cease following particular instructions, and pinpointing the cause of these issues often requires time.
The position influences how a model processes information, even if the content fits inside the context window. Researchers examining long-context retrieval have reported that content at the start or end of a long prompt receives more weight than content in the middle. This phenomenon is commonly referred to as "lost in the middle". In practice, a model is more prone to miss a specific fact when it appears on page forty of a ninety-page document than when it appears on page one or page ninety.
We saw this problem with a client's internal search tool that was built to answer questions using an onboarding handbook. During testing with short questions and a short handbook, the tool worked correctly. When the handbook grew to cover three departments, the tool started inserting the whole document into each query. Responses about policies in the middle of the document turned vague or inaccurate, while answers concerning the first and last sections stayed precise. The data still fit inside the context window, yet the model failed to spread its attention evenly over the text.
Why more tokens costs more than you'd think
Providers apply separate rates for input tokens and output tokens. Due to this pricing structure, the cost rises according to the amount of text supplied. If you double the input text amount, the processing cost also doubles before the model produces a response.
The recurring turn that you do not notice
What people often overlook is that a conversational system usually resends the entire history on each turn, since the model lacks its own memory between calls. If you ask a tenth question in a long chat, you are not paying for just one question. You are paying to reread the previous nine questions, their answers, and the new one.
A discussion that started at low cost can become costly simply due to its length, even when no new information is added.
The computational load grows as the token count rises. In a transformer, the attention mechanism compares each token with every other token, causing cost and latency to increase faster than linearly as sequences become longer. When the input length is doubled, the work required becomes more than twice as large. Some providers provide prompt caching, lowering these costs by charging less to reprocess a prefix the model has already processed. This is helpful for stable system prompts or documents that are queried many times. However, caching only reduces the price of the operation; it does not alter the fundamental scaling behaviour of the technology.
Why a larger window size is not always better
The most common way to address these issues is to use a model that has a larger context window.
That approach solves some problems but makes other issues more severe.
Larger context windows permit more irrelevant information to appear alongside the needed answer. This additional text incurs a cost even when it stays within the technical limits. The model must process, evaluate, and discard this information, which raises the risk of errors. If you supply a full product manual to answer a question about a single setting, you make the search more difficult. The model is more likely to add unrelated details to its response when that content seems relevant to the actual answer.
A larger context window does not improve a model's accuracy. It provides more space for an imprecise prompt to exist.
Large context windows lead users to submit very long prompts. A long prompt often shows that the user has not decided which information is most relevant. Providing all available data is easier than picking specific details, but this method often yields vague or incorrect answers. It also raises the cost of the query.
Retrieval: giving the model only what it needs
Rather than placing all information into the prompt, you can fetch only the relevant slice of it at query time and use it to generate an answer. This approach is called retrieval-augmented generation, or RAG. It is the typical solution when "our documents don't fit in the window" and the document set contains more than a few files.
The process is simple. Documents are split into segments, and each segment is turned into an embedding, a numerical representation of its content. These embeddings are then saved in a vector database. When a user poses a question, that question is transformed into an embedding using the same method. The database finds the segments whose meanings are most similar. Only these particular segments, usually a few paragraphs rather than the whole document collection, are sent to the model together with the question. The model does not access the entire set of documents. It only receives the parts marked as relevant, giving a smaller and more specific context for its response.
When implemented correctly, this approach keeps costs consistent as the knowledge base grows. The volume of data sent to the model per query stays the same even if the total library size doubles. If implemented poorly, the point of failure merely shifts. If the retrieval process picks incorrect data segments, the model will give a fluent response based on wrong evidence. No error message is produced for this occurrence. The output looks correct until a person checks the information.
Relying solely on semantic search of embeddings is insufficient, and depending on it alone is a common mistake. Meaning-based matching works well for finding similar concepts expressed in different language, but it struggles with exact data like product codes, order numbers, or specific clause references. A hybrid approach that combines embedding search with keyword matching and then reranks the results generally yields better outcomes than using a single method alone. This is particularly true for the precise queries that business users usually need.
Chunking is the process that determines retrieval effectiveness
The quality of retrieval depends on the text segments the system accesses. Choosing how to split a document into these segments, called chunking, is the hardest part of building a RAG system. This process is often regarded as a minor technical detail. In practice, it is the main factor that decides whether the system works correctly.
Too small loses the point, too big loses the precision
Dividing text into very small pieces, like single sentences or low token counts, causes the resulting segments to lose their meaning when taken out of context. A clause that needs the preceding paragraph for clarity will be retrieved by itself and will probably confuse the language model. Dividing text into very large pieces, such as whole sections or documents, makes the retrieval process return excessive information. Most of that text will be irrelevant to the specific question, which defeats the purpose of a retrieval system.
The usual practice is to split text into chunks of several hundred tokens each. These chunks should overlap a little to make sure that information near a boundary appears in both segments. A more effective approach is to split the text according to its natural structure, like headings, sections, or paragraphs, rather than using a fixed token count. Chunks that follow the document's original organisation are more likely to stay coherent and self-contained during retrieval.
Metadata does more work than people expect
Removing a chunk of text from its source causes loss of important context. You lose the document name, the section title, and the information's date. You should attach this metadata while chunking. This lets you filter search results by date or trace an answer back to its specific source paragraph. This trace is needed when you must verify why a model gave a specific response. If you omit metadata, each chunk becomes an anonymous paragraph. You will have no way to tell apart an outdated policy from 2019 and a current version.
Document rot is a data issue rather than a modelling issue. If a source library holds three versions of the same policy because old files were not archived, the chunking and embedding process will include all three. The model will retrieve the version that best matches the user's question. It will not give priority to the current version. The model will produce an answer that sounds authoritative even when the information is outdated. Cleaning the source library before chunking is more effective than trying to tune the retrieval process later.
Where to start
Determine your specific requirements before choosing a model based solely on window size. If your task requires reasoning across a single document, a long-context model is the right tool. You do not need to build a retrieval system for a single fifty-page contract. If you need to answer questions using a knowledge base that grows over time, you should use retrieval from the beginning, regardless of the current size of that data.
Count tokens on actual documents rather than guessing from clean samples. Messy documents are where estimates usually fail.
Then look honestly at your chunking.
Retrieve ten responses from your system and review the particular documents that were retrieved for each response. You will probably discover that numerous wrong answers stem from inadequate retrieval rather than a model error, even if the model appears confident.
None of this needs solving in one sitting, and it rarely gets solved well by adding more tokens to the budget and hoping. If you are weighing up whether your AI project needs a retrieval layer, or you are already running one and not sure why the answers still miss, talk to us.
From idea to deployment in weeks rather than quarters: our bespoke AI development applies articles such as this to your data.
Frequently asked questions
What is a context window?
A context window represents the full amount of text that a model can handle in a single pass. The limit is expressed in tokens rather than words. It encompasses the system prompt, any attached documents, the conversation history, and the response the model is presently generating. When a request goes beyond this limit, the system will either reject the input or, in a chat setting, delete or summarise earlier conversation sections without informing the user.
Which LLM has the largest context window?
This guide avoids ranking particular models since context window sizes shift with every release and differ across providers. How a model uses its context budget matters more than the sheer token count. A bigger window permits more irrelevant text to stay in the prompt together with the needed information, so a larger size does not inherently boost a model's precision or accuracy.
How do you increase a language model's context window?
It is not possible to enlarge the context window of an individual model. Instead, you need to choose a model built with a larger window or cut down the data you supply. A more efficient approach is retrieval, which means supplying only the pertinent sections of your documents when querying. This technique keeps costs from rising as your knowledge base expands.
Why does Claude's context window sometimes feel small?
Different providers employ varying context window sizes. A smaller window isn't automatically a drawback, as a larger window can let more irrelevant text stay close to the needed information. Whether a window size is sufficient hinges on if a task needs all data processed at once or if retrieval methods would be more effective irrespective of capacity.
How large is GPT-5's context window?
Exact numbers differ by provider and change with each model release, therefore this guide emphasises the budget itself rather than any single model's figure. Each token counted for a model includes the system prompt, attached documents, conversation history, and the output collectively, which is more important for reliability than whichever number currently leads the list.