AI evals

Sefism.AI evals

Tauseef Fayyaz

Chapter 02

Golden datasets

Where good cases come from, what each case needs and how to keep the set honest as it grows.

Golden dataset

Also called golden set, eval set, test set

What: A versioned collection of inputs with the expected answer, or the rules a good answer must follow, chosen to represent real use.
Why: It is the ruler every later change is measured with, so its quality caps the quality of every decision.

Where cases come from

What: Real traces and support tickets first, then bug reports, known hard cases, and cases written by domain experts.
Why: Real wording is messier than anything you invent, and that mess is what breaks apps.

Case format

What: An id, the input, the expected output or rubric, optional context (expected sources, allowed tools) and tags for the case type.
Why: Tags let you see that a change helped "policy" questions but hurt "billing" ones.

JSON

{"id": "bill-07", "input": "Why was I charged twice?",
 "expect_tool": "lookup_invoices",
 "rubric": ["explains the duplicate", "no refund promise"],
 "tags": ["billing", "multi-step"]}

Coverage

What: Common cases, each failure type you have seen, edge cases (empty, very long, other languages), out-of-scope requests and adversarial ones.
Why: A set of only happy-path questions scores 95% while users hit the other 5% daily.

Should-refuse cases

Also called negative cases, unanswerable questions

What: Inputs where the right behaviour is to say "I do not know", decline or ask a clarifying question.
Why: Without them nothing ever tests whether the app makes things up.

Synthetic cases

What: Cases generated by a model, for example questions written from your documents, then reviewed by a person.
Why: Useful to fill gaps quickly, but unreviewed synthetic cases are often too easy and too similar to each other.

Watch out

Letting the same model write the questions, answer them and grade them.

Keeping eval cases out of prompts

Also called leakage, contamination

What: Eval cases must not appear as few-shot examples, in fine-tuning data or pasted into the system prompt.
Why: Otherwise the eval measures memory of the answer, not the skill.

Versioning the set

What: Keep the dataset in Git or a dataset tool with versions, and record which version every run used.
Why: Scores from different dataset versions cannot be compared.

Go deeper