You shipped an LLM feature to production. Users are poking at it, a stakeholder wants to know whether it actually works, and your honest answer is that you are not quite sure. Manual spot checks eat an afternoon and still miss the failure that matters. This is the gap OpenAI Evals is built to close.
Evaluation is the difference between improving a system and just changing it. Without a measurement you trust, every prompt tweak and model swap is a guess, and you tend to find out it went wrong only when a customer tells you. We build and productionise AI systems at Shipshape Data, and the same pattern turns up on almost every engagement: teams ship something clever, then have no repeatable way to prove it works or to catch the day it stops working.
This guide walks through how to use OpenAI Evals to test and tune LLMs once they are live. You will set goals worth measuring, build a dataset that reflects real usage, wire evals into your deployment pipeline, and reuse a few patterns that save hours. None of it is exotic. The hard part is doing it before you need it rather than after an incident.
Why evaluation matters for production LLMs
An LLM that behaves perfectly in a demo can fall apart the moment real users arrive. Real production environments send it edge cases, half-finished sentences, ambiguous requests and inputs nobody on your team thought to try. Without a way to measure what happens, you cannot put a number on how often it fails, you cannot see the shape of those failures, and you certainly cannot prove that last week's fix helped.
The cost of skipping it
Manual checking feels responsible and is quietly dangerous. An engineer reviews ten outputs, they all look fine, everyone moves on, and the system is failing on fifteen percent of real queries that nobody sampled. Then someone edits the prompt to patch one complaint and introduces three new problems downstream, because there was no way to see the knock-on effect. Each release becomes a coin toss. Complaints arrive before your monitoring does, and confidence drains away because you cannot show anyone that things are getting better.
We have watched this play out. A client had a support classifier everyone was happy with until we built a proper test set from their real ticket history. It was misrouting one in six tickets, and two of the "obvious" prompt fixes the team had shipped earlier had each made accuracy worse, not better. Nobody knew, because nobody was measuring.
What measurement gives you
Structured evaluation turns opinion into evidence. You write down what good looks like for your business, assemble a test set that mirrors how people really use the system, and run automated checks that flag a regression the moment it appears. Once that is in place you can say, with data behind you, which prompt performs better, which model version reduces errors, and which edge cases still need work. The team ships because it can see the impact of a change, not because it feels brave.
Without measurement, an improvement and a random change look exactly the same from the outside.
What OpenAI Evals actually is
OpenAI Evals is an open-source framework for testing language-model outputs against expected results. You give it a set of input and output pairs, it runs your model over the inputs, and it scores how well the real outputs matched what you wanted. The framework handles the plumbing of running tests, comparing results and writing reports, which leaves you to do the only part that needs judgement: deciding what good performance means for your use case.
The three moving parts
There are three pieces to understand, and they fit together simply. The first is a dataset file in JSONL format, where each line is one test case with an input and the ideal output you expect. The second is a configuration file in YAML that names the eval, picks a scoring method and points at the dataset. The third is the command-line tool that runs the whole thing and hands back metrics: overall accuracy, where it failed, and what it produced instead.
How the scoring works
Each test sample holds a prompt your system would really receive and the response it ought to give. The framework sends every prompt to the model, captures the answer, and compares it against your ideal using whichever method you chose. Match scoring checks for an exact string match, which suits classification and extraction where there is one right answer. Includes scoring checks that the output contains the content that has to be there, which is more forgiving for free-text answers. Model-graded scoring uses a second LLM to judge quality when there is no single correct wording, as with a summary or an explanation.
Results land in structured JSON: an overall score, a per-sample breakdown, and the detail of each failure. That detail is the point. You read it to find patterns, work out which kinds of input trip the model up, and decide what to fix first. Test, read, adjust, run again. That loop is the whole method, and it compounds.
Step one: define goals, metrics and a golden set
You cannot measure what you have not defined. Before a single test case exists, you need to say what success means for your system and how you will know you have it. This step shapes everything after it, which is why it belongs to the people who understand the business problem, not only the engineers who understand the model. An eval that measures the wrong thing is worse than none, because it hands you false confidence.
Start from the business outcome
Write the job your system does in one plain sentence. For a support chatbot, "resolve tier-one queries without a human handover" is a goal you can measure, because it gives you a number: the escalation rate. "Answer questions accurately" is not, because nobody can agree where the line sits. The outcome you pick decides which behaviours you test and which mistakes you can live with. A tool that drafts marketing copy can tolerate a lot of variation. A tool that reads figures off an invoice cannot.
Then list the decision points where the model has to be right. A ticket triager might categorise the issue, judge how urgent it is, and route it to the right team. Write down every judgement the system makes and mark the ones that move the business. That list is your map for the test cases you build next.
Pick metrics that track the outcome
Choose measures that speak to the outcome rather than ones that look good on a slide. Accuracy tells you what share of outputs matched your ideal, but it will hide serious failures if your test set does not resemble real traffic. In practice you will usually reach for one of four. Exact match, when there is a single correct format. Includes, when the answer has to contain a specific fact. Fuzzy match, when the output and the reference need to overlap heavily without being identical. Model-graded, when several answers are all fine and only judgement can separate them.
Build a golden set worth trusting
Assemble fifty to a hundred input and output pairs that represent what your system meets in production. Pull them from real interactions, support tickets and internal workflows rather than inventing tidy examples, because the tidy ones are exactly the cases that already work. Each pair needs the input the system receives and the ideal output it should return, and your domain experts have to sign off on every ideal answer, since they are the ones who know what right looks like.
Then salt in the edge cases: rare, awkward and expensive when mishandled. If the system processes refunds, add the ambiguously worded request, the one missing half its information, the genuinely unusual circumstance. Weight the set toward the inputs that turn up most often and the failures that hurt most. Get that balance right and your evals catch both the everyday problems and the rare disasters before a user does.
Step two: build the dataset and config
The golden set is the raw material. Getting it right is the same job as any AI and ML data readiness exercise: real, representative, and reviewed by people who know the domain. To run it through OpenAI Evals you need two files: a JSONL dataset holding the test cases, and a YAML config that says how to score them. These two connect your business requirements to the framework, and getting them right is what makes the tests measure what you actually care about.
Write the JSONL dataset
Turn each golden-set example into a single-line JSON object in a file ending in .jsonl. Every object carries an input array with your prompt messages and an ideal field with the expected answer. The input array uses the same shape you already use when calling the OpenAI API: system and user messages that set context and pose the query. A support-triage case, for instance, has a system message telling the model to categorise tickets by urgency and department, a user message describing a failed payment, and an ideal of something like "URGENT - Billing".
Save the file with a name that tells you what it is, such as support-triage-eval.jsonl, and keep every golden-set example in it, one per line. Each line is a test the framework runs on its own, so adding a new edge case later is a matter of appending a line. That is deliberately easy, because you will be doing it often as production throws up new failures.
Write the YAML config
The config file tells the framework which scoring method to use and where the dataset lives. It names the eval, lists the metrics you want tracked, names the class that implements the scoring, and gives the path to the JSONL file. You place it in the evals registry directory inside the OpenAI Evals repository. The class you name is the decision that matters: Match for exact string comparison, Includes when the output has to contain specific text, or ModelBasedClassify when you need a second model to judge quality. Pick the one that matches how your domain experts would mark the same answers by hand.
Match the structure to the task
Different tasks want different shapes. Closed tasks like classification and extraction sit well with exact match. Open tasks like summarising or explaining need model grading, because there is no single string to compare against. A few pairings are worth keeping in mind: classification and data extraction both take a system-plus-user input and score with Match, against a single category string or a JSON object rendered as a string. Question answering usually takes the same input shape but scores with Includes against the answer text. Content generation takes the same input and scores with ModelBasedClassify against an example response. For model-graded evals you add a grading spec to the YAML that defines the criteria, and the framework uses a strong model you nominate to decide whether an output passes. That flexibility is what lets you evaluate tasks where several answers are all correct.
Step three: run evals and wire them into your workflow
The files are ready, but nothing is tested until you run the evals and fold the results into how you ship. Running them by hand gives you immediate feedback on a change. Building them into your pipeline stops a regression reaching production in the first place. The aim is to turn evaluation from a thing you remember to do into a gate that changes cannot get past.
Run from the command line
You run an eval with the oaieval command that ships with the framework. It takes two arguments: the model you want to test and the eval identifier from your YAML config. From the OpenAI Evals directory you point it at, say, a GPT-4 class model and your support-triage eval, and it loads the config, reads the dataset, sends each input to the model and compares the answers against your ideals. You watch progress tick past and get an accuracy score at the end. Adding a max-samples flag to cap it at ten runs a quick subset first, which is worth doing to confirm the config works before you spend money on the full sweep.
Results come back in the terminal and in a JSON log the framework writes to a temporary directory, printing the path when it finishes. That log holds the detail you came for: which samples failed and exactly what the model said instead of the right answer.
Read the failures, not just the score
Open the JSON and look past the headline number at the individual samples. Each one shows the input, the model's output, the ideal you expected, and whether it passed. Hunt for patterns rather than treating every miss as a one-off. You might find the model reliably fumbles requests with ambiguous wording, or falls over whenever a query mentions two issues at once. That is far more useful than knowing accuracy sits at eighty-three percent, because it tells you what to change.
Sort the failures into a small taxonomy: wrong category, wrong urgency, incomplete answer, and so on. Count how often each type happens and estimate what it costs when it reaches a user. That ranking decides your order of work, and it gives you a clean way to show improvement as you iterate on prompts or move to a different model.
Put it in the pipeline
Add an eval run as a required step in your continuous integration workflow, so every change triggers the tests automatically. Configure the pipeline to run evals before a pull request merges and to block anything that drops accuracy below a threshold you set. A simple job that runs the eval and then checks the score against a minimum is enough to catch a regression at review time instead of in production. This is where evals earn their keep, because the failure you never ship is far cheaper than the one you roll back.
Then schedule nightly runs against your live prompts on the current model versions, so you notice drift before a customer does. Store each run's accuracy in a database or even a spreadsheet, and you build a history that shows plainly whether the system is getting better or quietly sliding. Model providers change their models underneath you, and a nightly eval is how you find out that yesterday's upgrade shifted your behaviour.
Patterns worth reusing
The framework ships with tested patterns for the evaluation problems that come up again and again. Treat them as templates: copy the structure, change the inputs and the scoring to fit your domain, and you have a working eval in minutes rather than an afternoon.
Validating generated code
Checking code generation means asking whether the model produced something that both parses and solves the problem. For text-to-SQL, the pattern feeds the model a schema and a natural-language question, then uses model grading to judge the result, because two different queries can answer the same question correctly. A test case gives the system a schema for orders and customers, asks for orders over a thousand pounds from last month, and holds a reference query as the ideal. You score it with ModelBasedClassify and a grading spec that tells the judge model to check both syntax and logic, so a query that runs but answers the wrong question still fails.
Tracking multi-turn conversations
Production chatbots hold conversations, and the quality of a reply often depends on what was said three turns ago. Build test cases that carry the conversation history in the input array, so you can check whether the model keeps track and uses earlier information correctly. A case might include the system prompt, a user message giving an order number, the assistant's reply that the order shipped yesterday, and a follow-up asking when it will arrive. When several replies are acceptable, make the ideal field an array of valid answers and let the framework match against any of them. That mirrors how real conversations run, where "within three to five working days" and "by Friday" are both fine.
Where to take it from here
You now have a way to measure LLM performance that catches failures before your users meet them. Start small: build one eval for your most important workflow, run it weekly, and widen coverage as new failure patterns surface. The practice gets stronger every time you feed it a fresh edge case from your production logs, because the test set slowly comes to look like reality instead of a whiteboard.
OpenAI Evals gives you the measurement. Keeping a production system healthy still takes ongoing work as models change and the business moves. Track accuracy over time, dig into any drop the day it happens, and keep your golden set current as your domain experts sharpen their sense of what good output is. Every run produces data that guides your next prompt change, your next model choice and what you build next.
Getting evaluation right is often what separates an AI pilot that impresses in a demo from one that survives contact with real users. If you want a hand building evals that fit your own use cases, or you are trying to move a promising pilot into a service you can rely on, talk to us and we will start from what good actually looks like for your system.