Two models currently lead every public leaderboard. When you test them against your own documents and specific edge cases, one model fails after fifty examples. This difference between leaderboard rankings and your specific requirements shows that evaluation is not a preliminary task to complete before releasing a model. Evaluation is the primary work of building the system.
To evaluate large language models, you must first identify the specific component you are testing. You need to determine if you are assessing the base model, a combination of a prompt and a model, or an entire software pipeline. After you define the scope, select metrics that correspond to your specific task. Create a dataset that represents the actual traffic the system will receive. You should use public benchmarks to calibrate your results, but do not rely on them as your final conclusion.
Most organisations that work with Shipshape Data have already completed a proof of concept using a large language model. These demonstrations were successful.
When a stakeholder asked for specific performance data, the only available information was a collection of manually reviewed examples. This guide outlines the process we use with clients to address that gap. It covers how to select appropriate metrics for a task, use benchmarks accurately, and create a reliable dataset. It also explains how to implement release gates to prevent problematic prompt changes from reaching production.
Identify the specific goals of your testing
Define your measurement goals before you select metrics or benchmarks. Most evaluation projects fail because the scope was never clearly established, rather than because of flaws in the scoring method. These failures occur when the dataset, the methodology, and the success criteria are based on conflicting assumptions held by different team members.
Select the layer you want to test
An LLM almost never sits on its own in production. It sits inside a system: a prompt template, a retrieval step, an output parser, guardrails. Any one of those pieces can fail independently, so name the layer you're testing before you write a single example. In practice you're usually testing one of four things:
- The foundation model on its own: this is raw capability without any customisation. It is the type of test used to compare GPT-4o, Claude, or Gemini using a zero-shot prompt.
- A prompt plus model combination: the way a model performs when using a specific prompt template, for example when a summarisation prompt is used on support tickets.
- A fine-tuned model: a model that has been adjusted using your own data, such as a model trained on internal compliance documents.
- The full pipeline: retrieval, prompt, model and parser working together, the shape you get with a RAG system answering questions against a document store.
Testing the wrong component can lead to misdiagnosis of the problem. For example, you might test the pipeline when the issue is with the model, or test the model when the problem is with retrieval. Prompt design can significantly affect scores-by as much as twenty points-even with the same model. Therefore, when comparing foundation models, use the same prompt for both. When stress-testing a RAG system before launch, test the entire pipeline and measure retrieval and generation quality as separate steps.
If you evaluate the wrong layer, your results will appear accurate while concealing the specific location of the failure.
Use a precise name for the task type
Different evaluation approaches are needed for classification, extraction, summarisation, open question answering and code generation. A common error is to use a single, general accuracy check for all of these tasks. This results in a single number that appears clear but provides very little useful information.
- Classification. The model assigns a label like sentiment, intent, or category. You should measure precision, recall, and F1 score using a labelled test set.
- Extraction. The model identifies specific entities or values within the text. You should compare these extracted segments against your ground truth annotations.
- Summarisation. The model compresses content. Judge fluency, coverage and whether it stays factually consistent with the source.
- Generation and Q&A. The model creates a free-form response. Evaluate the response for relevance, accuracy, and how well the source material supports it.
- Code generation. The model writes executable code. Run functional tests against expected outputs instead of evaluating the code by reading it.
- Conversation and dialogue. The model maintains context throughout multiple turns. You should evaluate whether the task is completed based on the entire session rather than judging each response individually.
Match the test to production conditions
A summarisation model that handles short news articles well can fall apart on dense legal contracts, because what the model sees at inference shapes what it can do. Before you build a single evaluation example, write down the typical length, format, language and domain of your real inputs, then ask what the model will receive in production and what downstream systems will rely on.
Decide what good looks like before you run anything
Evaluation without an agreed pass mark is just observation. You can run every benchmark going and still end up arguing about whether 87% is good, because nobody wrote down what good meant before the number existed.
Write numeric acceptance criteria
Define a specific target for every metric based on your business requirements before you begin testing. A statement like "The model should be accurate" does not provide enough information to make decisions. A requirement such as "Factual correctness must clear 90% on the validation set, scored by LLM-as-judge against ground truth" provides a result that you can use. Apply this same process to latency, refusal rates for sensitive topics, and task completion. Document these targets before you review any results. This prevents you from changing your standards after a model fails to meet its initial expectations.
Map the risks before they become incidents
Your primary metric is not the only factor that can cause a deployment to fail.
If you ignore a risk during the evaluation phase, it will likely cause a production incident later, often at an inconvenient time. You should identify the failure modes for your specific use case, such as factual hallucinations, inconsistent responses to paraphrasing, or failures in tone and compliance. Create a testing strategy to address each of these issues.
Constraints require the same careful consideration as other requirements. The cost of inference per call, maximum latency, data residency, and the approved list of providers should be considered alongside your accuracy targets. A model with excellent scores that exceeds your budget by a factor of three is not a viable solution.
Build a dataset you'd trust in an argument
Evaluation results depend entirely on the quality of the test data. If a dataset consists only of ideal or selected examples, the resulting performance scores will be high but the model will fail when it encounters real user traffic. You must use a dataset that represents the full variety of inputs the model will actually receive.
Pull from production, not from imagination
If you have a live system, your logs are your most useful data. Collect a sample that represents various query types, lengths, and topics. Do not edit or clean the data before using it. You must include poorly phrased or incomplete inputs in the set, as the model needs to demonstrate that it can process these examples.
If you are building a new product without existing logs, create manual examples based on the categories from the first step. You should include difficult cases such as out-of-scope questions, contradictory instructions, and requests that require a refusal. A dataset based on predicted user behaviour differs from a dataset based on actual user behaviour. This difference often leads to inaccurate confidence in evaluation results.
Label with a rubric two people can agree on
Every example needs a ground truth answer or reference an evaluator can score against, otherwise correctness is a matter of opinion. For generation you need a reference answer and a rubric defining what counts as correct, partially correct and wrong. Log the input, the reference output, the category and any known failure mode against every example.
Check that two individuals assign identical scores to the same output. Run a calibration session involving at least two labelers prior to finalising the dataset. When labelers disagree on an example, it indicates that the rubric lacks sufficient specificity. Subsequent scores you gather will mirror this inconsistency.
Pick metrics that fit the task, not the leaderboard
There is no single metric that works for all tasks. Relying on a familiar metric such as accuracy frequently results in optimising for an incorrect outcome. Once you determine your task type, pick the metric that directly assesses the requirements of that job.
In classification, precision, recall, and F1 give a dependable assessment of a model's label assignments, particularly when class sizes differ. Extraction applies the same principles by evaluating the exact match rate and token-level F1 relative to the ground truth. Overall accuracy should not be used for these tasks. A model might reach a 90% accuracy rate by predicting the majority class yet miss the specific category you need to monitor.
Generation tasks need particular evaluation tools. ROUGE measures word overlap between a generated output and a reference text, offering a fast metric for summarisation. BERTScore employs a semantic approach by comparing embeddings. This enables it to detect correct paraphrases that ROUGE could penalise. Use ROUGE for quick checks and BERTScore when word-for-word matching would incorrectly grade a valid response.
Open Q&A and long-form generation are hard to evaluate since they do not confirm content accuracy. Employing an LLM as a judge is the most practical approach for large-scale assessment. Score dimensions such as correctness, relevance, and tone separately rather than merging them into a single overall score.
Retrieval and generation must be measured separately in RAG systems. Context precision indicates whether the retrieved chunks contain the answer. Context recall shows whether all relevant information was retrieved. Faithfulness assesses if the answer stays within the provided context. Answer relevancy checks if the output addresses the specific question. A single aggregate score for these four metrics will not reveal which layer is causing errors.
Use benchmarks for calibration, not as the answer
Public benchmarks give a reference point beyond your own dataset. They are useful for comparing scores to a known standard. Nevertheless, you should not assume that a leaderboard score represents a final conclusion. A model may attain a high score on MMLU yet remain unsuitable for your support desk. A high score on a general benchmark does not reveal how a model will perform for your specific use case.
Choose a benchmark that matches the specific capability you need. A support assistant relies on its ability to follow instructions and deliver accurate facts. A coding assistant depends on whether the code it generates works correctly. These are the benchmarks you should be aware of:
- MMLU assesses general knowledge across 57 subjects. It is employed to gauge performance in general question answering and knowledge retrieval.
- HumanEval checks whether generated code works as expected, making it a suitable choice for tools used by developers.
- TruthfulQA measures the frequency at which a model generates false answers that appear plausible. This metric is relevant for compliance tasks.
- MT-Bench tests multi-turn instruction-following, relevant for conversational agents.
- HellaSwag tests commonsense reasoning and context completion.
Set a baseline before you compare anything
A baseline allows you to interpret a score. Without this reference point, a result of 87% on an evaluation set lacks meaning. You can use your current production system, a smaller open-source model, or a rule-based fallback that does not use a large language model as your baseline.
Record the complete baseline configuration before you begin testing. This includes the model version, the prompt template, the temperature setting, and all post-processing steps. If you change any of these variables between the baseline and candidate runs, the comparison will be invalid regardless of how clear the final data appears. Save this configuration in version control alongside the dataset. This prevents accidental edits that would void all subsequent results.
Build the evaluators: automated scoring and LLM as judge
After you select the metrics, you must build the system that generates the scores. The decisions you make during this process determine whether the pipeline remains reproducible and maintainable in six months.
What to automate
Automated scoring is fast, inexpensive, and consistent. It is the appropriate tool for tasks that use deterministic calculations. You can implement string-based metrics such as exact match and token F1 directly. Connect the scorer to your CI pipeline to generate a new score automatically every time you change a prompt.
Where LLM-as-judge earns its place
LLM-as-judge is the most effective method for generation tasks that have multiple valid answers. You should send each output to a separate evaluator model with a structured rubric. The model should provide a score and the reasoning for that score. If you use a vague prompt like "score this response on quality", the judge will produce inconsistent scores that are not useful for evaluation.
Create the rubric to evaluate one dimension at a time. Score factual correctness on a defined scale of wrong, partially correct, or fully correct. Return this result as structured data instead of a paragraph that requires manual parsing. Follow the same process separately for relevance and tone. This method ensures that a poor result identifies the specific part of the system that failed.
RAG, tools and multi-turn conversations need their own tests
Single-turn evaluation does not identify many of the errors that occur in production. When you add retrieval, tool calls, or conversation history, each component introduces new potential failure points. A single aggregate score at the end of the process cannot identify which specific part of the system caused a failure.
Scoring RAG properly
A RAG pipeline requires testing for the retriever and the generator. You should score these components separately before you evaluate the system as a whole. Context precision measures whether the retrieved chunks contain the answer, while context recall measures whether the system retrieved all relevant information. Faithfulness measures whether the answer is based only on the retrieved content. This helps identify hallucinations that occur even when the retrieval step is successful. You should log each metric individually. If you see a high faithfulness score alongside low context recall, it indicates that the model produces grounded answers but the retriever is not providing enough evidence.
We spend the majority of our time on this part of evaluation with clients. This is where RAG systems frequently fail without obvious errors. A system may perform well during a demo with three specific questions but fail on the fortieth real enquiry. This happens because retrieval quality for disorganised production documents is a different problem than retrieval quality for the small, organised set used in initial testing.
Tool calls and multi-turn conversations
When a model calls a search tool, a calculator or an API, check whether it picked the right tool and formed the call parameters correctly, not just whether the final answer looked reasonable. A plausible final answer can hide a broken tool call that happened to land on a correct result by coincidence. This matters most where hallucination risk is already high: a broken tool call and a hallucinated answer can look identical from the outside.
For multi-turn systems, measure task completion and whether the model holds onto earlier context across the whole session rather than scoring each response alone. Build test sessions with at least five turns, include a topic shift, and add a follow-up question that only makes sense if the model remembers what was said earlier. A model that scores well turn by turn but loses the thread by turn three will frustrate a real user within ninety seconds.
Keep it honest: reproducibility, live feedback and CI gates
Evaluation results that cannot be reproduced are worse than having no results at all, as they provide unearned confidence. If a test suite produces different scores on different days without any changes to the system, the output is noise rather than a useful signal.
Pin your variables
The model version, temperature, top-p, prompt template, and dataset version all affect your score. If any of these variables change between runs, you cannot compare the results. Store the complete configuration as a versioned file alongside your dataset. Set the temperature to zero and use a fixed seed if the provider supports it. These steps will eliminate most of the noise between runs.
Some providers do not guarantee identical outputs for the same prompt even when the temperature is set to zero. If you cannot eliminate this variation, run each example three to five times. Calculate the average scores and track the standard deviation. If a model becomes less consistent between versions, this will affect your users even if the average score remains stable.
Monitor the results following the launch
Offline evaluation shows how a model performs with a set dataset. It does not show how people will actually use it, including uses the dataset did not cover. Session abandonment, a user restating a question, or a response being copied and significantly changed before use are examples of responses that did not meet user needs. Record each of these instances and link it to the response that came before it.
Add a simple, direct rating to each response-a thumbs up or thumbs down, with a space to briefly explain the rating-displayed immediately after the response. Make providing feedback easy, as complicated processes are often skipped. Review the negative ratings each week and categorise them by the type of problem: inaccurate information, missing information, or inappropriate tone. This collection of categorised failures will be your most valuable resource for improving prompts, because it is based on real-world performance.
Put it in CI so it actually happens every time
Include your evaluation suite as a continuous integration job that executes each time a pull request modifies a prompt, a model configuration, or a retrieval component. Handle a failed evaluation as you would a failed unit test, preventing the merge until the score reaches your defined threshold. Record the results of each run so you can monitor scores over time and detect performance regressions promptly.
Set strict requirements for your most critical metrics, which usually comprise faithfulness and factual correctness. Permit less critical metrics, like tone consistency, to generate warnings instead of halting a release. If you fail to enforce a hard block on your primary metrics, the team may disregard the results when pressured to meet a deadline. Log the scores and configuration ID for each release. This logging lets you pinpoint which specific change caused a production issue if one appears later.
You do not have to make this perfect on the first day.
Start by finishing the first two steps. Document the task definition, the success criteria, and the constraints before applying a benchmark. This avoids the need to rebuild the dataset during testing due to scope changes.
At Shipshape Data, we devote much of our time to assisting clients in defining performance standards for their systems before they invest in specific models. If you need an assessment of your current evaluation methods, talk to us. We will help you create a process that stays effective after you deploy your system to real users.
Considering building this for your business? Our AI developers will explain what it depends on, including whether your data is ready to support it.
Frequently asked questions
What are large language models?
Large language models are the systems that this evaluation framework is designed to test, and in production they almost never operate alone. They usually reside within a broader pipeline, a prompt template, a retrieval step, an output parser and guardrails, so specifying precisely which layer you are testing is important before you assess a metric.
How do large language models work?
You can test a large language model by assessing its raw capabilities, applying it with a fixed prompt, fine-tuning it on specific data, or embedding it in a retrieval-augmented generation pipeline. The particular configuration you select determines how you define success and which metrics yield accurate results.
What metrics should you use to evaluate a large language model?
The appropriate metric changes depending on the task. Precision, recall, and F1 scores work well for classification and extraction tasks. ROUGE or BERTScore are appropriate for generation tasks. Open-ended answers generally need LLM-as-judge scoring, where each aspect such as correctness and tone is evaluated individually.
How should you use benchmarks when evaluating an LLM?
Public benchmarks such as MMLU, HumanEval, or TruthfulQA give a reference point for comparing scores to established standards. However, a leaderboard result does not decide whether a model fits your specific requirements. You should choose a benchmark that aligns with the capability you need and set your own baseline before making any comparisons.