LLM Evals: How to Test AI Features With Golden Sets

A prompt change can fix one demonstration and quietly break a different customer workflow. Model output varies, provider behaviour changes and realistic inputs rarely resemble the clean examples used during development. Testing AI features therefore needs more than a handful of screenshots showing plausible responses.

Build an evaluation process around product decisions: whether a summary preserves the source, whether a classifier chooses the right category, or whether an assistant refuses an unauthorised action. The goal is not to prove that a model is generally intelligent. It is to collect enough evidence to decide whether a particular feature can ship.

Turn product requirements into observable checks

Start with the task contract. List the inputs your application accepts, the output it promises and the mistakes that would make the feature unusable. A support summariser might need to preserve dates and unresolved issues, while a design assistant might need to produce valid component properties without inventing unsupported options.

Separate deterministic checks from judgement. JSON parsing, required fields, allowed categories and forbidden network destinations can be tested directly. Whether a rewrite preserves a subtle qualification needs a rubric or human review. Do not ask a model judge to decide something ordinary application code can establish exactly.

Define severity before running tests. A formatting defect may permit a release with a tracked issue; disclosing another customer's content should block it. Keep release gates explicit so a strong average score cannot hide a critical failure. A perfect tone score cannot compensate for unsafe tool use.

The AI SDK implementation guide describes where these checks fit around generation. Tests should exercise the surrounding application boundaries too, rather than treating a raw model response as the whole product.

Build a golden set that reflects real work

A golden set is a curated collection of inputs with expected behaviour and enough context to judge the result. It does not always require one exact answer. For a summary, store required facts and prohibited inventions; for classification, store the accepted label and the reasoning behind ambiguous cases.

Include ordinary requests, difficult but legitimate requests and adversarial cases. Cover long inputs, missing fields, conflicting instructions, multilingual content and unsupported tasks where relevant. Avoid adding exotic edge cases while neglecting the workflow customers actually use most often.

Keep data provenance with each case: where it came from, whether permission permits its use, how it was sanitised and why it matters. Remove secrets and personal information before committing fixtures. Synthetic examples can fill coverage gaps, but they should not replace all evidence from realistic use.

Split development examples from a held-out set. If you tune a prompt repeatedly against every known case, the reported score becomes less informative. Version fixtures and record intentional changes to expected behaviour. The OpenAI evaluation guidance is useful for structuring evaluations, even when your application uses another provider.

Combine exact assertions with a clear rubric

Use exact assertions for machine-checkable constraints. Validate an output schema, check referenced identifiers against supplied records and confirm that a generated URL uses an allowed origin. Test side effects through mocks or isolated services, not through production credentials.

Use the regex tester to develop narrow checks such as detecting a forbidden placeholder in generated copy. Test both matching and non-matching examples before moving the expression into your test suite. Regular expressions do not establish factual accuracy, and a passing pattern should never stand in for semantic review.

A compact case definition can keep expectations reviewable:

{
"id": "summary-unresolved-refund",
"requiredFacts": ["refund pending", "receipt requested"],
"forbiddenClaims": ["refund completed"],
"criticalFailure": "invented financial outcome",
"rubric": "Preserves status without adding a resolution"
}

For rubric-based judging, define each score with concrete examples. A judge should identify evidence in the source and output, explain the failure category and allow an uncertain result. Keep the rubric focused on one dimension at a time rather than asking whether the answer is simply good.

If you use a model judge, calibrate it against human decisions. Blind candidate identities, randomise presentation order when comparing answers and review disagreements. Judge outputs are evidence with their own error rate, not an independent ground truth. Include the judge configuration in evaluation records.

Compare candidates without hiding variability

Run the baseline and candidate on the same cases under comparable conditions. Record prompt version, model configuration, application revision and relevant provider settings. If retrieval is involved, preserve the retrieved context or identify its snapshot so changing documents do not masquerade as prompt improvements.

Measure run-to-run variation by repeating important cases. A single successful output cannot demonstrate reliability for a high-risk action. Choose repetition counts according to consequence and cost, and report the number of attempts rather than suggesting certainty from a small sample.

Report results by slice as well as overall. Short English inputs can conceal poor performance on long documents or another supported language. Show counts and denominators, especially for small categories. Distinguish a true regression from a change whose evidence is too thin to interpret confidently.

Track latency and cost beside quality, without collapsing them into one unexplained score. The latency workflow helps separate first useful output from total duration. Use the NIST AI Risk Management Framework to frame documentation and risk review beyond a single benchmark number.

Make regression checks part of delivery

Run cheap deterministic checks on every relevant change. Reserve larger model-backed suites for candidate releases, scheduled checks or high-impact modifications. Provider calls can be slow and variable, so give the evaluation runner explicit budgets, retry policies and failure reporting.

Separate infrastructure failures from model failures. A timeout, exhausted quota or broken fixture should not silently become an incorrect answer or disappear from the denominator. Mark the run incomplete when necessary, and rerun under a documented policy rather than repeatedly trying until a candidate passes.

Use production feedback to extend coverage after privacy review. A user correction is a lead for investigation, not automatically the right reference answer. Reproduce the failure, establish expected behaviour and add the smallest case that protects against recurrence.

For safety-sensitive workflows, include hostile retrieved content, misleading tool results and requests for prohibited actions. The OWASP guidance on prompt injection explains why normal instruction-following tests are insufficient. Keep human review for consequential release decisions and maintain a rollback path when live behaviour differs from the test set.

Key takeaways

• Test specific product promises rather than general model quality.
• Curate versioned cases with provenance, expected facts and failure severity.
• Combine deterministic assertions with calibrated human or model judgement.
• Compare repeated runs and important input slices, not just averages.
• Gate releases on critical failures and expand coverage from reviewed incidents.