How to use OpenAI Evals to test and tune LLMs in production

You have deployed an LLM feature to production. Users are interacting with it, a stakeholder asks if it truly works, and your candid response is that you are not entirely certain. Manual spot checks consume an afternoon yet still overlook the critical failure. This is the gap that OpenAI Evals aims to close.

To employ OpenAI Evals, follow three steps. First, define your success criteria, then build a JSONL dataset and a YAML configuration file for scoring. Execute the tool and incorporate it into your deployment pipeline. Examine the specific failures that arise and use the evaluation outcomes to approve or block each release.

Evaluation separates improving a system from merely changing it. Without a reliable measurement, each prompt tweak or model modification is a guess. You will probably only discover an issue when a customer reports it. At Shipshape Data we build and deploy AI systems, and we observe the same pattern in most projects. Teams launch sophisticated features but lack a repeatable method to demonstrate they work or to detect when they fail.

This guide describes how to use OpenAI Evals to test and improve large language models post-deployment. You will define specific metrics, construct a dataset from real user interactions, embed evaluations into your deployment process, and apply several efficient workflows; these methods are straightforward. The main challenge is implementing these systems before problems arise.

Why evaluation matters for production LLMs

Large language models may work correctly in demonstrations but fail when faced with real user traffic.

Real production environments contain edge cases, incomplete sentences, ambiguous requests, and inputs your team did not anticipate. Without a method to measure these events, you cannot quantify the frequency of system failures. You will also be unable to pinpoint the patterns of those failures or show that a recent fix was effective.

The expense of omitting it

Manually checking outputs appears responsible, yet it introduces hidden risks. An engineer could review ten examples that look correct, causing the team to move forward while the system fails on fifteen per cent of real queries not included in the sample. When someone changes a prompt to address a specific issue, they often generate three new problems because they cannot see the broader consequences of the change. This renders every release unpredictable. Customer complaints appear before internal monitoring detects issues. Confidence in the project wanes because there is no data to prove the system is improving.

We've watched this play out.

A client employed a support classifier that appeared effective until we built a test set from their actual ticket history. The system misrouted one out of every six tickets. Two prompt adjustments the team previously implemented had reduced accuracy rather than improving it. The team was unaware of these issues because they were not measuring performance.

What measurement gives you

Structured evaluation turns subjective opinions into objective evidence. You define the specific success criteria for your business, build a test set from actual user behaviour, and implement automated checks to spot regressions instantly. This process lets you use data to decide which prompts perform better, which model versions reduce errors, and which edge cases need more attention. The team releases updates because they can measure a change's impact rather than relying on intuition.

If you do not measure your results, you cannot tell a deliberate improvement from a random change.

An explanation of OpenAI Evals

OpenAI Evals is an open-source framework that tests language-model outputs against expected results.

You supply a collection of input and output pairs. The system executes your model on the inputs and rates how closely the actual outputs meet your requirements. The framework handles the technical tasks of executing tests, comparing results, and producing reports. This lets you concentrate on the subjective task of defining what counts as successful performance for your specific application.

The three moving parts

You must understand three components that operate together. The first component is a JSONL dataset file in which each line holds one test case with an input and the expected output. The second component is a YAML configuration file that sets the evaluation name, chooses a scoring method, and specifies the dataset location. The third component is a command-line tool that runs the evaluation and returns metrics such as overall accuracy, specific failures, and the actual outputs produced.

How the scoring works

Each test sample holds a prompt your system would really receive and the response it ought to give. The framework sends every prompt to the model, captures the answer, and compares it against your ideal using whichever method you chose. Match scoring checks for an exact string match, which suits classification and extraction where there's one right answer. Includes scoring checks that the output contains the content that has to be there, which is more forgiving for free-text answers. Model-graded scoring uses a second LLM to judge quality when there's no single correct wording, as with a summary or an explanation.

Results appear in structured JSON: an overall score, a per-sample breakdown, and details of each failure. That detail is the focus. You examine it to find patterns, determine which types of input trip the model up, and decide what to fix first. Test, read, adjust, and run again. That loop constitutes the entire method, and it compounds.

Step one: define goals, metrics, and a golden set

Measurement needs a clear definition. You must define what success looks like for your system and how you will identify it before creating any test cases. This step sets the direction of the entire project. Therefore, the people who understand the business problem must be involved, instead of leaving the task solely to the engineers who understand the model. An evaluation that measures the wrong metrics is harmful because it gives false confidence.

Begin with the business outcome

State the function of your system in a single plain sentence. For a support chatbot, "resolve tier-one queries without a human handover" is a measurable goal. This goal yields a specific metric, the escalation rate. "Answer questions accurately" is not a measurable goal because there is no consensus on defining accuracy. The outcome you select determines which behaviours you test and which errors are acceptable. A tool that generates marketing copy can permit significant variation. A tool that extracts data from an invoice must be precise.

List the specific points at which the model must make a correct decision. For example, a ticket triager categorises an issue, evaluates its urgency, and routes it to a specific team. Document each judgement the system makes and identify which ones directly affect business operations. Use this list as a guide for creating your test cases.

Select metrics that monitor the result

Select metrics that capture real outcomes rather than metrics that appear impressive in presentations. Accuracy reports the percentage of outputs that meet your ideal, yet it can hide major errors when your test data does not mirror actual user traffic. Typically, you will use one of four specific methods. Apply exact match when only one correct format exists. Apply includes when the response must contain a particular piece of information. Apply fuzzy match when the output and the reference should be similar but need not be identical. Apply model-graded evaluation when many different answers are acceptable and require qualitative judgement to differentiate them.

Build a golden set worth trusting

Gather fifty to one hundred input-output pairs that represent the real data your system processes in production. Use real interactions, support tickets, and internal workflows rather than simplified examples, because simplified cases typically already work correctly. Each pair must contain the system input and the correct output. Your domain experts must approve each ideal answer, as they define the correct standards.

Also add edge cases during testing; these are rare situations that can be costly if not handled correctly. If your system processes refunds, include requests that are unclear, lack complete information, or represent truly exceptional circumstances. Target the testing data towards the most common inputs and the failures that would cause the greatest damage. Finding the right balance will let your evaluations detect both common problems and rare critical issues before users encounter them.

Step two: build the dataset and config

The golden set is raw material. Preparing this data follows the same process as any other AI or machine learning data readiness task. The data must be real, representative, and reviewed by domain experts. To use OpenAI Evals, you need to supply two files: a JSONL dataset with the test cases and a YAML configuration file that defines the scoring methods. These files connect your business requirements to the framework. Properly configuring these files ensures the tests measure your specific objectives.

Write the JSONL dataset

Transform each example in your golden set into a single-line JSON object inside a.jsonl file. Each object must include an input array for your prompt messages and an ideal field for the expected response. The input array uses the standard OpenAI API format, employing system and user messages to set context and supply the query. In a support-triage example, the system message would tell the model to categorise tickets by urgency and department, the user message would describe a failed payment, and the ideal field would hold a value like "URGENT - Billing".

Give the file a descriptive name, such as support-triage-eval.jsonl, and place all golden-set examples with one example per line. The framework executes each line as an independent test. You can add new edge cases by appending lines to the file. This process remains simple because you will often update the file when new failures appear in production.

Write the YAML config

The config file informs the framework which scoring method to use and where the dataset resides. It specifies the eval name, lists the metrics to track, names the class that implements the scoring, and provides the path to the JSONL file. You store it in the evals registry directory within the OpenAI Evals repository. The class you select is the key decision: Match for exact string comparison, Includes when the output must contain specific text, or ModelBasedClassify when a second model is needed to assess quality. Choose the one that aligns with how your domain experts would manually grade the same answers.

Ensure the structure matches the task

Different tasks need different evaluation methods. Closed tasks like classification and extraction work well with exact match scoring. Open tasks such as summarising or explaining require model grading because no single correct string exists for comparison. There are several common pairings to remember. Classification and data extraction use a system-plus-user input and are scored with Match against a category string or a JSON object. Question answering uses the same input format but is scored with Includes against the answer text. Content generation uses the same input and is scored with ModelBasedClassify against an example response. For model-graded evaluations, you add a grading spec in the YAML file to define the criteria. The framework then uses a designated strong model to decide if an output passes. This flexibility lets you evaluate tasks where multiple different answers are correct.

Step three: run evals and wire them into your workflow

The files are ready, but testing is not complete until you run the evaluations and integrate the results into your deployment process. Running evaluations manually gives immediate feedback on specific changes. Including them in your automated pipeline prevents regressions from reaching production. The goal is to shift evaluation from a manual task to a requirement that all changes must satisfy.

Run this from the command line

You execute an eval using the oaieval command that comes with the framework. It requires two arguments: the model you wish to test and the eval identifier from your YAML config. From the OpenAI Evals directory you specify, for example, a GPT-4 class model and your support-triage eval, and it loads the config, reads the dataset, sends each input to the model, and compares the answers to your ideals. You watch the progress advance and receive an accuracy score at the end. Adding a max-samples flag to limit it to ten runs a quick subset first, which is useful to confirm the config works before spending money on the full sweep.

The results show up in the terminal and in a JSON log file that the framework writes to a temporary directory, showing the file path when finished. This log includes the detailed information you requested, such as which samples failed, the model's output, and the expected correct answer.

Read the failures, not just the score

Open the JSON and look beyond the headline number at the individual samples. Each sample displays the input, the model's output, the ideal you expected, and whether it passed. Search for patterns. Do not treat every miss as a one-off. You may find the model consistently fumbles requests with ambiguous wording, or fails whenever a query mentions two issues at once. That is far more useful than knowing accuracy is eighty-three per cent, because it tells you what to change.

Organise the failures into categories like wrong category, wrong urgency, or incomplete answer. Count how often each error type occurs and calculate the cost of these errors reaching a user. Use this ranking to prioritise your tasks. This method offers a clear way to show progress as you update prompts or switch to different models.

Add it to the pipeline

Include an evaluation run as a mandatory step in your continuous integration workflow. This makes certain that each change automatically triggers the tests. Configure the pipeline to execute evaluations before a pull request merges and to reject any code that drops accuracy beneath your defined threshold. A straightforward job that runs the evaluation and checks the score against a minimum value will detect regressions during review. This stops errors from reaching production. Avoiding a failure before release is cheaper than rolling back a deployment.

Set up nightly tests for your active prompts on current model versions to spot performance changes before customers notice them. Log the accuracy of each test in a database or spreadsheet. This builds a history that indicates whether the system is improving or declining. Model providers frequently update their models. Running nightly evaluations lets you detect when an update alters your application's behaviour.

Patterns worth reusing

The framework contains tested patterns for common evaluation problems. You may use these as templates by copying the structure and tailoring the inputs and scoring to your specific domain. This lets you build a functional evaluation quickly.

Validating generated code

To verify code generation, you need to determine whether the model produced a response that is syntactically correct and solves the specific problem. In text-to-SQL tasks, the process consists of giving the model a schema and a natural-language question. You then apply model grading to assess the output, since several different queries can yield the correct answer to the same question. For instance, a test case could supply a schema for orders and customers, ask for a list of orders over one thousand pounds from the previous month, and include a reference query as a baseline. You can score it using ModelBasedClassify together with a grading specification. This specification directs the judge model to check both syntax and logic. This makes a query fail if it runs successfully but answers the wrong question.

Tracking multi-turn conversations

Production chatbots engage in conversations, and the quality of a reply often relies on what was said three turns earlier. Create test cases that include the conversation history in the input array, allowing you to verify whether the model tracks and uses earlier information correctly. A case could contain the system prompt, a user message with an order number, the assistant's reply stating the order shipped yesterday, and a follow-up asking when it will arrive. If multiple replies are acceptable, set the ideal field to an array of valid answers and allow the framework to match any of them. That reflects real conversation flow, where "within three to five working days" and "by Friday" are both acceptable.

Next steps

You now possess a method to measure LLM performance and spot errors before they reach your users. Start by creating a single evaluation for your most important workflow. Execute this test weekly and expand your coverage as you uncover new failure patterns. The process improves as you add edge cases from your production logs to the test set, ensuring the evaluations reflect actual usage.

OpenAI Evals supplies the necessary measurements. Keeping a production system running demands continuous effort as models evolve and business requirements shift. Monitor accuracy trends, investigate performance drops promptly, and update your golden dataset as domain experts refine the criteria for high-quality output. Data from each run guides your subsequent prompt adjustments, model selections, and development priorities.

Effective evaluation separates an AI pilot that performs well in a demo from one that works for real users. If you need help building evaluations for your specific use cases, or if you aim to turn a pilot into a reliable service, talk to us. We will start by defining the specific requirements for your system.

From concept to production in weeks, not quarters: our bespoke AI development applies articles like this to your data.

Frequently asked questions