How does an evaluation set work as a validation artefact in regulated work?

It is what makes a non-deterministic system validatable at all. You cannot specify that an LLM returns output Y for input X, so the specification becomes statistical: a frozen, versioned evaluation set with an acceptance threshold agreed before testing, and revalidation triggered by any change to the model version, prompt, chunking strategy or retrieval parameters. Three properties turn a working eval set into an auditable one: it is versioned in source control alongside the code, the threshold was set before results were seen, and every run is retained with the system version it tested. Without those, you have a quality tool but not evidence.

How to Build an LLM Evaluation Set That Survives Audit

Q: How many test cases does an LLM evaluation set need?

Fewer than most teams assume, and the number matters less than the composition. A few hundred well-chosen cases covering the real distribution of inputs plus known failure modes is more useful than several thousand generated ones, because every case has to be labelled by someone who can say what correct looks like - and that labelling effort is the actual constraint. Start with enough cases that a meaningful regression moves the score visibly, then grow the set by adding real failures as they occur rather than by generating volume up front.

Q: Where do the test cases come from?

Four sources, in descending order of value. Real production traffic, sampled to reflect the actual input distribution rather than the inputs you expected. Failures that already shipped, added the week they are found - this is the bucket that stops regressions and it compounds. Deliberate edge cases from the people who own the workflow, since they know which inputs are rare and consequential. And adversarial cases, including prompt-injection attempts. Synthetic generation fills gaps production has not yet exercised, but unreviewed synthetic cases dilute exactly the trust that makes the set worth having.

Q: Should you score retrieval and generation separately?

Yes, and it is the single most useful structural decision in RAG evaluation. A system that retrieves the wrong document and summarises it fluently fails for a completely different reason than one that retrieves correctly and overstates the conclusion. One combined score hides both, and points remediation in the wrong direction: teams tune prompts when the problem is chunking, because the number does not distinguish them. Score whether the right context was retrieved, then whether the answer is faithful to that context, and treat them as separate metrics with separate thresholds.

Q: How do you keep an evaluation set from going stale?

Treat it as a living asset with a light recurring routine rather than a project that finishes. Add real production failures weekly, while the context is fresh. Re-review labels when the policy or definition of correct changes, because a label that was right last quarter may not be. Prune cases that test behaviour you deliberately removed, or the set punishes you for intended changes. And watch for contamination: once a case has influenced prompt tuning, it measures memorisation rather than capability, which is why a small held-out portion the team does not iterate against is worth keeping.

Q: Do you need an evaluation tool to start?

No. The set is the asset and the tool is replaceable - which is the right way round, because teams routinely pick a platform first and end up with tool-shaped data they cannot move. A versioned file of inputs, expected outputs and metadata, plus a script that runs it and records results, is enough to catch regressions and is portable to whatever tooling you adopt later. Choose tooling when the volume of runs makes manual review the bottleneck, and prefer options with OpenTelemetry support so the traces remain yours.

The Short Answer

An evaluation set is a versioned collection of real inputs with agreed correct outputs, run on every meaningful change. It outlives your model choice, your framework and usually your vendor, which is why it - not the tooling - is the thing worth investing in. That argument is made in LLM observability and evaluation tools; this page is the how.

The part almost nobody writes about is the second job it does. In any regulated context, the eval set is the validation artefact - the only mechanism by which a non-deterministic system can be validated at all. Three properties separate a useful eval set from an auditable one, and they cost nothing if you build them in from the start.

What Goes In: Four Buckets

Composition matters far more than size. A few hundred well-chosen cases beat several thousand generated ones, because every case needs a label from someone who can say what correct looks like - and that labelling capacity, not case count, is your real constraint.

Four sources of evaluation cases, what each catches, and how it is maintained
Bucket	Catches	Maintenance
Sampled production traffic	Regressions on what users actually do	Resample when the input mix shifts
Shipped failures	The same bug twice - the bucket that compounds	Add weekly, while context is fresh
Edge cases from process owners	Rare, consequential inputs production has not shown yet	Review when the process changes
Adversarial cases	Injection and manipulation attempts	Grow as new attack shapes appear

Synthetic generation has a place: filling coverage production has not exercised yet. But unreviewed synthetic cases dilute precisely the trust that makes the set worth having, so anything generated needs the same human label as anything sampled. If nobody has time to review it, it does not belong in the set.

The shipped-failures bucket is the one that compounds. Every production failure added is a bug that cannot silently return. Teams that build only this bucket still end up with something valuable; teams that build everything except it keep re-fixing the same things.

Score Retrieval and Generation Separately

For a RAG system this is the highest-value structural decision, and getting it wrong sends remediation in exactly the wrong direction.

A system that retrieves the wrong document and summarises it fluently, and a system that retrieves the right one and overstates the conclusion, both produce a wrong answer and both score badly on a combined metric. They need completely different fixes - chunking and retrieval in the first case, prompting and grounding in the second. A single number cannot tell you which, so teams tune prompts while the real problem sits in the chunker.

Score two things separately: was the right context retrieved, and is the answer faithful to the context retrieved. Separate thresholds, separate trend lines. The published benchmark runs report them that way - 144 of 150 on retrieval, and 0 fabricated citations across 297 cases - because they answer different questions about the system.

The Version That Survives Audit

In regulated work the eval set stops being a quality tool and becomes evidence. Under 21 CFR Part 11 and equivalent regimes, it is the only way to validate a system whose output is not fixed: you cannot specify that the model returns Y for X, so the specification becomes statistical.

Three properties make the difference, and all three are free at the start:

Versioned in source control, alongside the code. A spreadsheet in someone Drive is not an artefact. The set that validated release 1.4 must be recoverable, exactly, after release 1.9.
The threshold was agreed before results were seen. Deciding what “good enough” means after looking at the numbers is how validation becomes theatre. Write it down, dated, before the run.
Every run retained with the system version it tested. Model version, prompt version, chunking and retrieval parameters. Otherwise you can show a score but not what produced it - and a reviewer is asking about the second thing.

Retrofitting these is the expensive path. Nothing reconstructs a threshold that was never recorded, and a set that was never versioned cannot be shown to be the set that validated anything.

Keeping It Honest

Building an eval set is a week. Keeping it representative for six months, while prompts, models and users all change, is the part most teams get wrong - and the failure is quiet: the set keeps returning good scores while measuring less and less.

Three specific decay modes:

Contamination. Once a case has been used to tune a prompt, it measures memorisation rather than capability. Keep a held-out portion the team does not iterate against, and treat a sudden jump on the full set with suspicion.
Drift. The input distribution moves as the product and its users change. A set sampled at launch stops representing reality within months. Resample periodically rather than assuming.
Stale labels. When policy or the definition of correct changes, previously correct answers become wrong. If nobody re-reviews, the set actively penalises intended behaviour and the team learns to ignore it - which is worse than having no set at all.

And one thing worth checking in a regulated or personal-data context: if the set was built from production traffic, it contains production data, and it lives outside the normal data lifecycle by design. That makes it the artefact most often missed in an erasure request.

Start Without a Tool

A versioned file of inputs, expected outputs and metadata, plus a script that runs it and records the result, catches regressions from day one and is portable to any platform you adopt later.

The failure mode of starting with a tool is ending up with tool-shaped data you cannot move. Choose tooling when the volume of runs makes manual review the bottleneck rather than at the outset, and prefer options with OpenTelemetry support so the traces remain yours - the argument in the tools comparison.

Frequently Asked Questions

How many test cases does an LLM evaluation set need?

Fewer than most teams assume, and composition matters more than count. A few hundred well-chosen cases covering the real input distribution plus known failure modes beats several thousand generated ones, because every case needs a label from someone who can say what correct looks like - and that labelling effort is the real constraint. Start with enough that a meaningful regression moves the score visibly, then grow by adding real failures rather than generating volume.

Where do the test cases come from?

Four sources: real production traffic sampled to reflect the actual input distribution; failures that already shipped, added the week they are found - the bucket that compounds; deliberate edge cases from the people who own the workflow; and adversarial cases including injection attempts. Synthetic generation fills gaps production has not exercised, but unreviewed synthetic cases dilute the trust that makes the set worth having.

Should you score retrieval and generation separately?

Yes - the single most useful structural decision in RAG evaluation. A system that retrieves the wrong document and summarises it fluently fails for a different reason than one that retrieves correctly and overstates the conclusion. One combined score hides both and points remediation the wrong way: teams tune prompts when the problem is chunking. Score whether the right context was retrieved, then whether the answer is faithful to it, with separate thresholds.

How does an evaluation set work as a validation artefact?

It is what makes a non-deterministic system validatable. You cannot specify that an LLM returns Y for X, so the specification becomes statistical: a frozen versioned set with a threshold agreed before testing, and revalidation triggered by any change to model, prompt, chunking or retrieval parameters. Three properties make it auditable - versioned in source control, threshold set before results were seen, and every run retained with the system version it tested. Without those you have a quality tool, not evidence.

How do you keep an evaluation set from going stale?

Treat it as a living asset. Add real production failures weekly while context is fresh. Re-review labels when policy or the definition of correct changes. Prune cases testing behaviour you deliberately removed, or the set punishes intended changes. And watch for contamination - once a case has influenced prompt tuning it measures memorisation rather than capability, which is why a held-out portion the team does not iterate against is worth keeping.

Do you need an evaluation tool to start?

No. The set is the asset and the tool is replaceable, which is the right way round - teams routinely pick a platform first and end up with tool-shaped data they cannot move. A versioned file of inputs, expected outputs and metadata plus a script that runs it is enough to catch regressions and portable to whatever you adopt later. Choose tooling when run volume makes manual review the bottleneck, and prefer OpenTelemetry support so traces remain yours.

The Asset You Keep

Models get deprecated, frameworks get replaced, vendors get acquired. The set of real inputs with agreed correct outputs survives all of it, and it is the only thing that lets you answer the question that actually matters when you change something: did that make it better or worse?

Build it small, version it properly, add every failure the week it happens, and record the threshold before you look. In regulated work those same habits are the difference between a system you believe works and one you can show works - which is the whole argument behind publishing the runs.