A support bot tells a customer the returns window is sixty days. The real policy is thirty. Nobody wrote a sixty-day rule anywhere the model could see it. It built a plausible number out of the shape of similar policies it had absorbed during training, and it delivered that number with exactly the same even confidence it uses for everything else it says. The refund goes through. Finance finds the gap three weeks later, when the numbers stop reconciling and someone finally asks where the sixty came from.
A large language model, an LLM, is software trained to predict the next piece of text in a sequence, at a scale where that narrow skill starts to look like understanding, reasoning and conversation. It has read an enormous amount of writing. It has not read your returns policy unless you put it in front of the model yourself, and it will not tell you that it hasn't. It will just answer, because answering is the one thing it always does.
We build data and AI systems at Shipshape Data, and this is close to the most common gap we walk into: a business that has bought or built something powered by an LLM without anyone in the room being able to say, precisely, what the model does and doesn't know, or what it costs to ask it things at scale. Enthusiasm arrives first. The mental model of how the thing actually works arrives later, usually after an incident. This guide covers what an LLM is doing under the surface, how tokens and context windows shape what it can hold in mind at once, where training comes from, what these models are genuinely good and bad at, what they cost to run, and how to choose and check one for a real business use case.
What is actually happening when you type a prompt
Strip away the interface and an LLM is a very large statistical function. Feed it a sequence of text and it returns a probability for every possible next word, or more precisely every possible next token, then picks one, appends it, and repeats. Do that a few hundred times and you get a paragraph that reads like a person wrote it. There is no lookup happening, no retrieval of a stored fact called "your returns policy". There is a probability distribution, shaped by everything the model saw during training, deciding what word is most likely to come next given everything written so far.
Not a database, not a search engine
This is the single most useful thing a business reader can hold onto. A database returns what was actually stored. A search engine returns documents that actually exist. An LLM returns whatever its training made statistically probable, which is usually close to true and sometimes is not, and the model has no internal flag that lights up to tell you which case you're in. It is exactly as fluent when it's wrong as when it's right. Nobody forgot to patch that behaviour. It's the mechanism working exactly as designed, pointed at a question it was never built to answer honestly.
Why it feels like reasoning
At small scale, next-word prediction produces gibberish. At the scale of modern models, trained on a meaningful fraction of everything ever written down and tuned on billions of parameters, the same mechanism produces text that plans ahead, holds a thread of logic across paragraphs, and answers questions it was never directly trained on. Nobody fully agrees on how much of that is genuine reasoning and how much is an extremely good imitation of the shape reasoning takes in text. For a business decision, the distinction matters less than the practical consequence: it works remarkably well on problems that look like the training data, and it degrades in ways that are hard to predict on problems that don't.
Tokens: why the model doesn't read words the way you do
A model doesn't see letters or whole words. It sees tokens, which are chunks of text, sometimes a whole word, sometimes a few letters, sometimes a single punctuation mark, decided by a fixed vocabulary built before training ever started. "Shipshape" might be one token or three, depending on whether the model's vocabulary happened to learn that exact string as a unit. Common English words are usually a single token. Rare words, made-up names, most non-English scripts and anything with unusual spelling tend to fragment into several.
Where this actually bites
Tokenisation is invisible until it isn't. A model can genuinely struggle to count the letters in a word it has never seen tokenised as a whole, because the underlying representation is a string of token identifiers, not letters, and that representation doesn't preserve spelling the way a human eye does. Product codes, serial numbers, and anything with a lot of digits and punctuation tend to tokenise unevenly, which is part of why models are unreliable at exact-match tasks like copying a long reference number without a single digit slipping. If your use case leans on precise strings, that is worth testing before you trust the output, not after.
Tokens also matter because every API call is priced and limited by them, input and output combined, so a wordy prompt with a long system instruction and a large pasted document costs real money before the model has said a single useful thing back.
Context windows: the memory that quietly runs out
The context window is the maximum number of tokens a model can hold in a single call, prompt and reply together. Once a conversation, or a document, exceeds it, the oldest content is dropped or never sent in the first place, and the model answers as if it were never there. It doesn't say "I've forgotten the first half of this document." It just answers from what remains, fluently, and the gap is often invisible until an answer contradicts something stated earlier in the conversation.
Bigger context is not a free upgrade
Modern models advertise context windows running into the hundreds of thousands of tokens, which sounds like it solves the problem outright. It helps, but it doesn't remove the underlying issue, which researchers sometimes call "lost in the middle": models are measurably better at using information near the start or end of a long context than information buried in the centre of it. Stuffing an entire knowledge base into the prompt because the window technically allows it tends to produce answers that quietly favour whatever happened to sit near the edges. A longer context window is genuinely useful. It is not a substitute for giving the model the right smaller set of facts, which is most of what retrieval-augmented generation exists to do: fetch the relevant handful of passages instead of hoping the model finds them in a haystack.
A model with a huge context window and the wrong document in it is still going to answer confidently. Size doesn't fix relevance.
Where the behaviour actually comes from
Training happens in stages, and each one shapes a different part of what you eventually talk to.
Pretraining
The first and by far the most expensive stage is pretraining: the model reads a vast amount of text, mostly scraped from the public web, books and code, and adjusts billions of internal parameters to get slightly better at predicting the next token, over and over, across a training run that can cost tens of millions of pounds in compute for the largest models. This is where the model picks up grammar, facts, style, and a rough shape of how the world is described in writing. It is also where it picks up whatever biases, gaps and errors exist in that writing, because the model has no separate mechanism for judging truth from confident-sounding fiction. It absorbed both.
Fine-tuning and human feedback
A raw pretrained model is a strange conversational partner: it completes text, it doesn't naturally answer questions or refuse harmful requests. The stage that turns it into an assistant is fine-tuning, usually followed by reinforcement learning from human feedback, where people rank the model's responses and the model is nudged towards the ones humans preferred. This is where "helpful, harmless assistant" behaviour comes from. It is also where a lot of a vendor's specific personality and guardrails get baked in, which is why the same underlying capability can feel noticeably different between providers.
The knowledge cutoff
Because pretraining is a snapshot of text gathered up to a certain date, every model has a knowledge cutoff, a point after which it knows nothing unless something in your prompt tells it. Ask about an event from after that date and a well-behaved model should say it doesn't know. Plenty don't, and will instead generate something plausible-sounding for the period it wasn't trained on, which is one of the more dangerous failure modes because there is nothing in the phrasing that flags it.
What LLMs are genuinely good at
Language tasks, first and foremost. Summarising a long document, rewriting something in a different tone, drafting a first pass at an email, extracting structured fields from messy free text, translating, classifying support tickets by topic. These play directly to the mechanism: predicting plausible, well-formed language given some input, and doing it fast across a volume no team of people could match.
They are also genuinely useful as a flexible front end onto other systems, when wired up carefully. A model that turns a plain-English question into a structured query, or that reads a document and hands back the three fields you actually need, is doing something a rules-based parser would take far longer to build and would break the moment the input format shifted slightly. That flexibility is the real selling point, and it's a fair one.
Where teams get the most reliable mileage is narrowing the job. A model asked to summarise a specific document in front of it is on solid ground. A model asked open questions about the world at large, with no source material supplied, is working from memory and probability alone, and that is where the next section starts.
Where they still fall over
Hallucination
The returns-policy example at the top of this guide has a name: hallucination, when a model states something false with the same fluent confidence as something true. It is not lying, in the sense of knowing the truth and choosing otherwise. It genuinely has no separate channel for "I am now guessing" versus "I am now certain." Both come out the same way. This is the single most important thing to design around, and grounding the model in real retrieved documents rather than asking it to answer from memory is the main lever that helps.
Arithmetic, exact counting, and anything brittle
Models are surprisingly patchy at maths, particularly multi-step arithmetic on larger numbers, precisely because they are predicting plausible next tokens rather than executing a calculation. They can get a sum right for reasons that have nothing to do with actually computing it, and wrong for the same reasons. The fix in production is rarely "wait for a smarter model": it's giving the model a calculator, a code interpreter, or a proper backend to call, and using the model to decide when to call it rather than to do the sum itself.
Consistency and determinism
Ask the same question twice and you can get two different answers, even with settings turned down to reduce randomness. That is workable for a chat assistant and awkward for anything that needs the same input to reliably produce the same output, such as automated decisioning. If a use case needs that kind of consistency, plan for it explicitly rather than assuming a model call behaves like a function call.
Cost and latency: what actually shows up on the invoice
Every token in and every token out is billed, usually per million tokens, and the price differs by an order of magnitude or more between the largest frontier models and their smaller, faster siblings. A long system prompt repeated on every call, a habit of pasting whole documents instead of the relevant paragraph, and a chat history that keeps growing every turn all quietly compound into a bill that looks nothing like the number in the demo. We have seen the same pattern more than once: a prototype that cost pennies to test scales to production volumes and the bill turns out to be the single biggest surprise in the whole project, not the model quality.
Latency is the other constraint people underestimate. Bigger, more capable models are usually slower to respond, and a chain that calls a model two or three times to check its own work, which is a completely reasonable thing to do for quality, multiplies that wait. A customer-facing chat interface can tolerate a couple of seconds. A background process extracting data from ten thousand documents overnight can tolerate far more, and using a smaller, cheaper, faster model there rather than defaulting to the flagship is usually the right call, not a compromise.
The practical habit worth building is matching model size to task difficulty rather than reaching for the most capable model available by default. A lot of classification and extraction work runs perfectly well on a fraction of the cost of the frontier option, and the savings scale with volume in a way that matters the moment a pilot becomes a real workload.
Where to start
Start with the task, not the model. Write down, specifically, what you want the assistant to do, what information it's allowed to see, and what a wrong answer actually costs you: an annoyed customer is a different order of risk to an incorrect number on a regulatory filing. That single sentence usually rules out half the shortlist before you've opened a single pricing page.
Then test on your own examples, not the vendor's. Build a small set of real inputs, including the awkward ones your business actually produces, and check the outputs by hand before you check them with another model. Our guide on how to evaluate large language models goes into the metrics and benchmarks worth using once you're past that first manual pass, but the manual pass has to happen first. A benchmark score has never once told us what a specific model does with a specific client's messy invoices.
Ground it before you trust it. If the model needs to know something specific to your business, put that thing in front of it at query time rather than hoping training absorbed it, and treat prompt engineering as the cheap, fast lever it is before reaching for anything heavier. Most disappointing pilots we get called in to fix were asking a general-purpose model to know things it was never shown, and then being surprised when it guessed.
None of this requires picking the single best model in the world. It requires knowing what your model can see, what it costs at your volume, and what happens the day it's confidently wrong. If you want a straight read on which of those gaps you're carrying right now, talk to us.