Two models sit at the top of every public leaderboard this quarter. Put them against your own documents and your own awkward edge cases, and one falls apart within fifty examples. That gap, between what a leaderboard says and what your system needs, is why evaluation is not homework before shipping an LLM. It is the job.
Most of the organisations we work with at Shipshape Data have already run a proof of concept with an LLM. The demo went well. Then someone asked how well, exactly, and nobody had an answer beyond a shrug and a spreadsheet of examples somebody eyeballed once. This guide is the process we walk clients through: picking metrics that fit the task, using benchmarks without being misled by them, building a dataset you can defend, and gating a release so a bad prompt change never quietly reaches production.
Work out what you're actually testing
Before you touch a metric or a benchmark, decide exactly what you are measuring. Most evaluation efforts that go wrong do not fail because the scoring method was flawed. They fail because nobody agreed on the scope, so the dataset, the method and the pass mark got built on assumptions three different people were carrying around in their heads.
Pick the layer under test
An LLM almost never sits on its own in production. It sits inside a system: a prompt template, a retrieval step, an output parser, guardrails. Any one of those pieces can fail independently, so name the layer you are testing before you write a single example. In practice you are usually testing one of four things:
- The foundation model on its own: raw capability with no customisation, the kind of test you run comparing GPT-4o against Claude or Gemini on a zero-shot prompt.
- A prompt plus model combination: how the model behaves against a fixed prompt template, such as a summarisation prompt applied to support tickets.
- A fine-tuned model: one adapted on your own data, for instance trained on internal compliance documents.
- The full pipeline: retrieval, prompt, model and parser working together, the shape you get with a RAG system answering questions against a document store.
Get this wrong and you test the pipeline when the real question was about the model, or test the model when the actual bug lives in retrieval. Prompt design alone can swing a score by twenty points on the same underlying model, so if you are comparing two foundation models for a future build, hold the prompt identical across both. If you are stress-testing a RAG system before go-live, test the full pipeline and measure retrieval and generation quality separately.
Evaluating the wrong layer gives you results that look correct and hide exactly where the failure lives.
Name the task type precisely
Classification, extraction, summarisation, open Q&A and code generation all need different evaluation approaches. The common mistake is running one generic accuracy check across all of them, which produces a number that looks tidy and tells you almost nothing.
- Classification. The model assigns a label such as sentiment, intent or category. Measure precision, recall and F1 against a labelled test set.
- Extraction. The model pulls entities or values out of text. Compare the extracted spans to your ground truth annotations.
- Summarisation. The model compresses content. Judge fluency, coverage and whether it stays factually consistent with the source.
- Generation and Q&A. The model produces a free-form answer. Judge relevance, correctness and how well the answer is grounded in a source.
- Code generation. The model writes executable code. Run functional tests against expected outputs; do not evaluate by reading the code.
- Conversation and dialogue. The model holds context across turns. Judge whether the task gets completed across the whole session, not each response on its own.
Match the test to production conditions
A summarisation model that handles short news articles well can fall apart on dense legal contracts, because what the model sees at inference shapes what it can do. Before you build a single evaluation example, write down the typical length, format, language and domain of your real inputs, then ask what the model will actually receive in production and what downstream systems will rely on.
Decide what good looks like before you run anything
Evaluation without an agreed pass mark is just observation. You can run every benchmark going and still end up arguing about whether 87% is good, because nobody wrote down what good meant before the number existed.
Write numeric acceptance criteria
Translate the business requirement into a specific target for each metric before you test. "The model should be accurate" settles nothing. "Factual correctness must clear 90% on the validation set, scored by LLM-as-judge against ground truth" gives you a result you can act on. Do the same for latency, refusal rate on sensitive topics, and task completion. Write these down before you see a single result, because it is far too easy to move the goalposts once a model has underperformed.
Map the risks before they become incidents
Your primary metric is not the only thing that can sink a deployment. A risk you skip at evaluation time tends to show up as a production incident six months later, usually at the worst moment. Go through the failure modes specific to your use case, hallucination on domain facts, inconsistent behaviour under paraphrasing, tone or compliance misses, and write a test strategy against each one.
Constraints deserve the same discipline. Inference cost per call, maximum latency, data residency and the list of providers you are allowed to use should sit alongside your accuracy thresholds. A model that scores brilliantly and costs three times your budget is not a real option.
Build a dataset you'd trust in an argument
Your evaluation is only as good as what you test it against. A dataset built from tidy, cherry-picked examples produces a score that looks strong in the deck and collapses within a week of real traffic. You want a dataset that reflects the actual spread of inputs the model will face.
Pull from production, not from imagination
If you already have a system live, your logs are the best raw material you own. Pull a representative sample across query types, lengths and domains, and resist the urge to tidy it up first. The badly phrased, incomplete inputs are exactly what the model needs to prove it can handle, so they belong in the set.
If you are building something new with no logs to draw on, generate examples by hand against the categories from the first step, and deliberately include stress cases: out-of-scope questions, contradictory instructions, requests that should trigger a refusal. A dataset built from what you imagine users will ask is a different dataset from what they actually ask, and that gap is where most evaluation confidence turns out to be false.
Label with a rubric two people can agree on
Every example needs a ground truth answer or reference an evaluator can score against, otherwise correctness is a matter of opinion. For generation you need a reference answer and a rubric defining what counts as correct, partially correct and wrong. Log the input, the reference output, the category and any known failure mode against every example.
Then check whether two people scoring the same output actually agree. Run a short calibration session with at least two labellers before the dataset gets finalised. If they diverge on the same example, the rubric is not tight enough, and any score you get later will carry that disagreement baked in.
Pick metrics that fit the task, not the leaderboard
No single metric works across every task, and reaching for the familiar one, usually accuracy, tends to optimise for the wrong thing. Once you have named your task type, map it to the metric that actually measures what matters for that job.
For classification, precision, recall and F1 give a balanced read on how well the model assigns labels, particularly when the classes are not evenly distributed. Extraction works on the same logic: exact match rate and token-level F1 against ground truth. Avoid overall accuracy for either, because a model can post 90% by exploiting a class imbalance while quietly failing on the category you actually care about.
Generation tasks need a different toolkit. ROUGE measures the overlap in wording between an output and a reference, a fast signal for summarisation. BERTScore takes a semantic approach instead, comparing embeddings so it recognises a correct paraphrase that ROUGE would penalise. Use ROUGE as a quick check and BERTScore when surface wording alone would unfairly mark down a valid answer.
Open Q&A and longer generation stretch both past their limits, because neither captures whether the answer is actually true. LLM-as-judge is the practical option at scale here. Score each dimension, correctness, relevance, tone, on its own rather than collapsing them into a single number.
RAG systems need retrieval and generation measured apart. Context precision asks whether the chunks you pulled back contained the answer, context recall asks whether you pulled back everything relevant, faithfulness asks whether the answer stays inside what the context said, and answer relevancy asks whether it addressed the question actually asked. A single aggregate score across all four hides which layer is causing the trouble.
Use benchmarks for calibration, not as the answer
Public benchmarks give you a reference point outside your own dataset, useful for calibrating scores against a known standard. The trap is treating a leaderboard score as though it settles the question. A model can top MMLU and still be the wrong choice for your support desk, because a high score on a general benchmark says nothing about your specific use case.
Match the benchmark to the capability you need. A support assistant lives or dies on instruction-following and factual accuracy; a code assistant lives or dies on whether the code it writes runs. The benchmarks worth knowing:
- MMLU tests broad knowledge across 57 subjects, useful for general Q&A and knowledge retrieval.
- HumanEval tests whether generated code is functionally correct, the obvious pick for developer tooling.
- TruthfulQA tests how often a model produces a false but plausible answer, which matters for compliance work.
- MT-Bench tests multi-turn instruction-following, relevant for conversational agents.
- HellaSwag tests commonsense reasoning and context completion.
Set a baseline before you compare anything
A baseline is what makes a score interpretable. Without one, 87% on your evaluation set means nothing. Your baseline can be your current production system, a smaller open-source model, or a rule-based fallback that handles the task without an LLM at all.
Write the full baseline configuration down before testing starts: model version, prompt template, temperature, and any post-processing. If any of those variables shift between the baseline run and the candidate run, the comparison is not valid any more, however clean the resulting numbers look. Store it under version control next to the dataset, so nobody can quietly edit it and invalidate every result that followed.
Build the evaluators: automated scoring and LLM as judge
Once the metrics are decided, someone has to build the machinery that produces a score. The choices made here decide how reproducible and maintainable the pipeline stays six months from now.
What to automate
Automated scoring is fast, cheap and consistent, the right tool for anything that maps to a deterministic calculation. String-based metrics like exact match and token F1 are simple to implement directly. Wire the scorer into your CI pipeline so every prompt change produces a fresh score automatically, rather than depending on someone remembering to run it by hand.
Where LLM-as-judge earns its place
For generation tasks where no single reference answer captures every valid response, LLM-as-judge is the most workable option. Send each output to a separate evaluator model alongside a structured rubric and have it return a score with reasoning attached. A vague prompt, "score this response on quality", produces inconsistent judge scores that are barely more useful than no evaluation at all.
Write the rubric to score one dimension at a time: factual correctness on a defined scale (wrong, partially correct, fully correct), returned as structured data, not a paragraph you then parse by eye. Do the same separately for relevance and tone, so a bad result points you straight at the part of the system that failed.
RAG, tools and multi-turn conversations need their own tests
Single-turn evaluation misses a large share of what actually breaks in production. Add retrieval, tool calls or conversation history, and each new component becomes its own way for the system to fail, so one aggregate score at the end will not tell you where the failure started.
Scoring RAG properly
A RAG pipeline has two things to test: the retriever and the generator, scored apart before you combine them. Context precision checks whether the chunks you pulled back actually contain the answer, context recall checks whether you pulled back everything relevant, and faithfulness checks whether the answer stays grounded in what was retrieved, catching a hallucination that slips through even when retrieval did its job correctly. Log each field separately: a high faithfulness figure next to low context recall tells you the model generates grounded answers, but the retriever keeps failing to surface the right evidence.
This is the part of evaluation we spend most of our time on with clients, because it is also where RAG systems most often go wrong quietly. A system can look fine in a demo with three well-chosen questions and still fail on the fortieth real one, because retrieval quality on messy production documents is a different problem to retrieval quality on the tidy handful you tested first.
Tool calls and multi-turn dialogue
When a model calls a search tool, a calculator or an API, check whether it picked the right tool and formed the call parameters correctly, not just whether the final answer looked reasonable. A plausible final answer can hide a broken tool call that happened to land on a correct result by coincidence. This matters most where hallucination risk is already high: a broken tool call and a hallucinated answer can look identical from the outside.
For multi-turn systems, measure task completion and whether the model holds onto earlier context across the whole session rather than scoring each response alone. Build test sessions with at least five turns, include a topic shift, and add a follow-up question that only makes sense if the model remembers what was said earlier. A model that scores well turn by turn but loses the thread by turn three will frustrate a real user within ninety seconds.
Keep it honest: reproducibility, live feedback and CI gates
An evaluation result you cannot reproduce is worse than no result, because it creates confidence that is not earned. If the same suite gives you a different score tomorrow with nothing changed, that is a noise problem, not a signal.
Pin your variables
Model version, temperature, top-p, prompt template and dataset version all move your score, and any one drifting between runs makes two results incomparable. Store the full configuration as a versioned file next to your dataset. Set temperature to zero and fix a seed wherever the provider allows it: that alone removes most of the run-to-run noise.
Even at zero temperature, some providers still do not guarantee identical output twice. Where you cannot eliminate that, run each example three to five times and average the scores, and track the standard deviation alongside the mean. A model getting less consistent between versions matters to your users even if the average score barely moves.
Watch what happens after launch
Offline evaluation tells you how a model performs against a fixed dataset. It cannot tell you how real people will use it in ways your dataset never anticipated. Session abandonment, a user rephrasing the same question twice, a response copied out and heavily edited before use: these all point at a response that fell short. Log each one against the response that preceded it.
Pair that with a lightweight explicit rating, a thumbs up or down with an optional reason, right where the response appears. Keep it light, because a clunky feedback flow gets ignored. Review the negative ones weekly and tag each by failure type: hallucination, missed information, wrong tone. That tagged pile becomes the most useful dataset you own for fixing prompts, because it is built from actual failures rather than ones you guessed at in advance.
Put it in CI so it actually happens every time
Add your evaluation suite as a CI job that fires on any pull request touching a prompt, a model configuration or a retrieval component, and treat a failed evaluation the way you would treat a failed unit test: it blocks the merge until the score clears the threshold you set earlier. Store the results from every run so you can plot scores over time and see a regression coming rather than discovering it from a support ticket.
Set a hard gate on the metrics that matter most, faithfulness and factual correctness usually belong here, and let softer ones like tone consistency raise a warning instead of blocking outright. A gate without a hard block on your most important metric is a suggestion, and suggestions get ignored the first time a release is running late. Log every release against its scores and its configuration ID, so when something goes wrong in production six weeks later you can trace exactly which change caused it.
None of this needs to be perfect on day one. Start with the first two steps: getting the task definition, the success criteria and the constraints written down before you touch a benchmark saves you from rebuilding the dataset halfway through testing because the scope shifted underneath you.
We spend a good chunk of our week at Shipshape Data doing exactly this with clients: working out what "good" actually means for a specific system before anyone commits budget to a specific model. If you want a second pair of eyes on where your own evaluation setup has gaps, talk to us and we will help you build a process that still holds up once real users get their hands on it.