Chapter 2 of 7 · 5 checks · 2 min
Eval dataset
The dataset decides what your eval can see. These checks make it representative, unambiguous and safe to keep.
Build the dataset from real or realistic inputs, not tidy examplesMust
Why it matters and how to check
OpenAI's guide lists datasets that do not reflect production traffic as an anti-pattern, and Anthropic's guide says evals should mirror the real task distribution. Draw cases from production logs, pilot users or domain experts, with permission and with personal data removed.
To verify, compare the eval set with a sample of real traffic: input lengths, languages, topics and the share of unusual requests should be broadly similar. If the eval inputs are all clean and short while real inputs are messy and long, the scores will flatter the system.
Include edge cases on purpose: empty, long, ambiguous, off-topic and hostile inputsMust
Why it matters and how to check
Anthropic's guide names irrelevant input, overly long input and ambiguous cases as edge cases to include. Add inputs where the right answer is a refusal, a clarifying question or "the information is not available", so the eval rewards the system for not guessing.
To verify, tag each case with a type and count them. Every edge-case type you care about should have several cases, and at least some cases should have "decline" or "not found" as the expected result.
Make each case unambiguous, with a reference answer or grading ruleMust
Why it matters and how to check
Anthropic's agent evals article gives a test: a good task is one where two domain experts would independently reach the same pass or fail verdict. It also recommends a reference solution that passes all graders, which proves the task is solvable and the graders are set up correctly.
To verify, give 20 random cases to two people and have them grade the same outputs independently. Cases where they disagree need a clearer expected answer or a sharper rubric before they count.
Start with a small set from real failures and grow it with every bugShould
Why it matters and how to check
You do not need hundreds of cases to begin. Anthropic's agent evals article says 20 to 50 simple tasks drawn from real failures is a great start. Anthropic's general eval guide adds that more cases with slightly noisier automated grading beat fewer, hand-graded ones, so grow volume once grading is automated.
To verify, check that each bug fixed in the last month added at least one eval case, and that the set's size and last-change date are visible in version control.
Version the dataset and keep a held-out slice you never tune againstShould
Why it matters and how to check
If you tune prompts against every case, the score stops measuring real quality. Keep a held-out slice that is only run before releases, and version the whole set so a score can always be traced to the exact data it came from.
To verify, confirm the dataset lives in version control or a versioned store, that each eval run records the dataset version, and that the held-out slice is excluded from day-to-day prompt iteration.