A New York lawyer once filed a legal brief full of citations to court cases that do not exist. The model had invented them, complete with case numbers and judges' names, and it did so in the same confident tone it used for the real citations sitting next to them. Nobody on the legal team checked, because nothing about the fake ones read as fake. That is the whole problem in one paragraph: the model is not lying, and it is not guessing in any way you can detect from the outside. It is doing exactly what it was built to do, and what it was built to do is not the same thing as being right.
Hallucination is what the industry calls it when a model states something false with the same fluency and structure as something true. It is not a bug in the sense of a line of code misbehaving. It is a consequence of how these systems are trained: to predict a plausible next word, not to check a fact against the world. A model with no source for an answer will still produce one, because producing an answer is what it does, and a well-formed sentence with a wrong number in it looks identical to a well-formed sentence with a right one.
We build data and AI systems at Shipshape Data, and hallucination is the failure mode that ends more pilots than any other, usually right at the point someone tries to put the system in front of a customer or a regulator. This guide covers why models produce confident wrong answers, how grounding a model in retrieved evidence reduces the problem without eliminating it, why narrowing the brief matters as much as the technology, what citations do and do not prove, how to build a verification step before an answer ships, how to measure factuality honestly, and how to design an interface that lets the user check the answer rather than simply trust it.
Why the model sounds sure when it is wrong
Large language models are trained to predict the next most likely word given everything that came before it. That objective produces remarkably fluent writing. It does not produce a mechanism for checking whether a claim is true, because truth was never the thing being optimised. The model learned patterns of language from an enormous body of text, and somewhere in that training data the pattern "a court case is cited like this" got learned perfectly well, independent of whether any particular case cited that way actually exists.
There is no internal fact-checker
People often assume a model has something like a memory it can consult, the way a person might picture a specific meeting before describing it. It does not work like that. Ask a model a question with no clear answer in its training data and it does not return "I don't know" by default, it returns the most statistically plausible continuation of the question, and a specific wrong number is exactly as plausible, structurally, as a specific right one. The model has no separate step where it checks the number against a source, because there is no source attached to the sentence at generation time. It is composing, not recalling, even when the output reads exactly like recall.
Confidence is a writing style, not a signal
This is the part that catches people out. A model that is unsure of a fact does not hedge more than a model that is sure of it, unless it has specifically been trained or prompted to hedge. Fluency and accuracy are produced by the same process and they do not move together. You cannot look at the tone of an answer and infer how likely it is to be true, which is exactly backwards from how we read confident human speech, and it is why a hallucinated statistic sits in a paragraph looking no different from the true one three lines above it.
Grounding: giving the model something to point at
The single most effective structural fix is retrieval-augmented generation, usually shortened to RAG. Instead of asking the model to answer from what it learned during training, you fetch the relevant documents at the moment of the question, put them in front of the model, and ask it to answer using only what is there. Our longer piece on vector databases covers the retrieval infrastructure behind this in more depth, but the principle worth holding onto here is simple: a model that is shown the answer is far less likely to invent one.
Retrieval reduces the problem, it does not remove it
Grounding narrows the space in which the model has to guess, and that narrowing does most of the work. It does not close the gap entirely. A model can still misread a retrieved passage, blend two documents into a claim that neither one makes on its own, or answer confidently from a chunk of text that was retrieved because it matched the wording of the question without actually containing the answer. Retrieval quality depends on the underlying search, and an index built on embeddings that were never checked against real user questions will surface plausible-looking passages that are quietly off topic. Garbage in at the retrieval stage produces confident garbage out at the generation stage, dressed up as grounded fact.
What actually goes wrong in practice
The failures we see most often are not exotic. A document gets chunked badly, so a table's header ends up in one retrieved fragment and its numbers in another, and the model happily reunites the wrong header with the wrong row. A policy gets superseded and the old version is still in the index because nobody deleted it, so the model retrieves it, cites it accurately, and gives an answer that was correct eighteen months ago. Retrieval fixes the model's imagination problem. It does not fix a stale or badly organised document store, and teams sometimes ship RAG expecting it to compensate for content that was never properly maintained in the first place.
Constraining the brief: the model should be allowed to say no
Grounding answers what the model can draw on. It does not answer what the model should attempt in the first place, and that is a separate, cheaper fix that gets skipped surprisingly often.
Scope is a design decision, not a technical limit
Left open-ended, a model will answer almost anything put in front of it, including questions its source material does not cover, because answering is the default behaviour and refusing has to be built in deliberately. The fix is to write that refusal into the system prompt and the surrounding product, explicitly: if the retrieved context does not contain the answer, say so, rather than filling the gap with something fluent. This sounds obvious and is routinely skipped, because a system that occasionally says "I don't have that information" reads as less impressive in a demo than one that always answers, and demos are where a lot of these decisions quietly get made.
Narrow the domain, narrow the risk
A support assistant scoped tightly to one product line will hallucinate less than one asked to field questions about the whole company, not because the model has been improved, but because the space of plausible wrong answers has shrunk. This is one of the least glamorous levers available and one of the most reliable. Every domain you add to the brief is a domain the model can now confidently get wrong, and the cost of that is rarely weighed against the convenience of a single assistant that answers everything.
Citations and why they need checking too
Asking a model to cite its sources feels like it should solve the problem outright. It helps, and it is not sufficient on its own, because a model can generate a citation the same way it generates any other plausible-looking text: fluently, and without any guarantee the thing it is citing says what the answer claims it says.
A citation is a claim, not proof
The Avianca case in the opening of this guide is exactly this failure. The citations looked correct in form: case name, reporter, page number, all in the right order. What was missing was any actual court case behind them. A citation generated by a language model needs the same scepticism as any other output, because the mechanism that produces a real citation and the mechanism that produces a fabricated one are the same mechanism.
A citation only proves a source exists if something checked that it does. Otherwise it is just a fact wearing a fact's clothing.
What a trustworthy citation pipeline looks like
The reliable version links every citation back to an actual retrieved passage, ideally with the exact span highlighted, rather than letting the model produce a reference from memory. If the citation cannot be traced to a real document your system actually pulled, it should not be shown as a citation at all. This is more engineering than it sounds. It means threading provenance through the whole pipeline, from the original chunk to the final rendered answer, so that "source 3" in the output is a genuine pointer rather than a plausible-sounding label the model attached after the fact.
Building a verification step before the answer ships
Grounding and citation both happen before the model writes its answer. Verification happens after, and it is the step teams are most likely to skip under deadline pressure, because it adds latency and it is not visible in a demo the way a fast, confident answer is.
Automated checks catch the easy cases
The cheapest version is a rule-based check: does every number in the answer actually appear in the retrieved context? Does every named entity match something in the source? These checks will not catch subtle misreadings, but they catch a surprising amount, because a lot of hallucination is crude rather than clever, a figure invented wholesale rather than a nuanced misreading of a real one.
A second pass, done properly, is worth the latency
A more thorough approach runs a second model pass whose only job is to check the first answer against the retrieved sources and flag anything unsupported. This is not free, it adds a call and a delay, and for some use cases that trade is not worth making. For anything answer-of-record, a figure in a report, a claim in a regulated document, a statement a customer will act on, it usually is. The mistake we see most is teams building this stage last, if at all, treating it as a nice-to-have polish rather than the step that actually decides whether the system is safe to ship.
Evaluating for factuality, not fluency
Most evaluation setups for language models measure whether the output reads well, whether it is relevant, whether it is roughly on topic. Factuality is a different axis and needs its own test, because a model can score well on fluency and relevance while being confidently wrong on the one number that mattered.
Build a test set of known answers
The most useful evaluation we build with clients is a set of questions with verified, human-checked answers pulled from the same domain the system will actually operate in, and it is run every time the model, the prompt, or the retrieval setup changes. Generic benchmarks tell you how a model performs in general. They tell you very little about whether it will get your specific product policy or your specific contract clause right, and that is the question that actually matters once the system is live. Our guide on using OpenAI Evals to test and tune LLMs covers one practical route into this kind of testing if you want to go further.
Track the rate, not the anecdote
A single wrong answer spotted by a nervous stakeholder tends to trigger a lot of activity and not much measurement. What actually matters is the rate: out of a representative sample of questions, how many answers contained an unsupported claim, and is that rate going up or down as you change things. Without a tracked number, every fix is a guess and every regression is invisible until a customer finds it.
Designing the interface so the user can check
Even a well-grounded, well-verified system will occasionally get something wrong, because none of the fixes above reach zero. The last line of defence is not a smarter model. It is an interface that makes checking the answer part of using the product, rather than something the user has to think to do on their own initiative.
Show the source, not only the answer
The simplest version of this is putting the retrieved passage next to the answer, not buried behind a link three clicks away. If a user has to go looking for the evidence, most of them will not bother, and the confident answer will stand unchallenged whether it deserved to or not. If the evidence sits right there, checking becomes a five-second glance rather than a chore, and users actually do it.
Let uncertainty show
A system that flags low-confidence retrieval, thin source coverage, or a question outside its stated scope, and says so visibly rather than papering over it with a fluent paragraph, earns more trust over time than one that always sounds equally sure. This overlaps with the wider discipline of responsible AI, though it is a narrower, more practical version of it: not explaining the whole model, just showing enough of the working for a user to sanity-check one specific answer.
Make correction part of the loop
Give users an easy way to flag a wrong answer, and actually route those flags somewhere a person looks at them. This does two things: it catches errors that slipped past every other check, and it builds the evaluation set described earlier, because a flagged wrong answer with the correct answer attached is exactly the kind of test case that makes the next version better. Teams that skip this treat every hallucination as a one-off embarrassment rather than a data point, and end up rediscovering the same failure mode every few months.
Where to start
Do not start by shopping for a hallucination-detection tool. Start by writing down what the system is actually supposed to know about, and building in the refusal for everything outside that boundary. That single decision removes more risk than most of the technical fixes that come after it, and it costs nothing beyond an uncomfortable conversation about scope.
Then ground it properly: retrieval built on content that is actually maintained, citations that trace back to a real passage rather than a plausible-sounding label, and a verification pass before anything ships to a user who might act on it. Build a small, honest test set from your own domain and watch the factuality rate over time rather than reacting to the odd anecdote. None of this is exotic engineering. Most of it is discipline that gets skipped under deadline pressure, which is exactly why it is worth doing properly the first time.
If you are building an AI system that has to be trusted with an answer someone will act on, and you are not confident it would survive someone actually checking its sources, talk to us. We would rather help you find the gaps before a customer or a regulator does.