Chapter 1 of 7 · 4 checks · 2 min
Success criteria and scope
An eval measures progress toward a goal, so the goal comes first. These checks make sure everyone agrees what good looks like before anything is graded.
Write specific, measurable success criteria for each featureMust
Why it matters and how to check
Anthropic's guide on building evals describes good criteria as specific, measurable, achievable and relevant. "Good answers" is not a criterion; "category matches the human label on at least 90% of the test set" is. Base targets on prior experiments, benchmarks or expert knowledge rather than on hope.
To verify, find the written criteria for the feature. Each one should name a metric, a threshold and the dataset it is measured on. If a criterion cannot be computed by a script or a defined human review, rewrite it.
Cover every dimension that matters, not only accuracyMust
Why it matters and how to check
Anthropic's guide notes that most use cases need evaluation along several criteria at once, listing dimensions such as task fidelity, consistency, relevance and coherence, tone and style, privacy preservation, context use, latency and price. A feature that is accurate but leaks personal data, or accurate but takes 40 seconds, has still failed.
To verify, list the criteria next to these dimensions and mark each as covered, deliberately skipped with a reason, or missing. Missing safety, privacy, latency or cost criteria are the most common gaps.
Agree a ship rule: which thresholds must pass before a change goes liveMust
Why it matters and how to check
Criteria only protect users if they block releases. Decide which metrics are hard gates, which can drop slightly with sign-off, and what counts as a regression, such as any drop greater than two points on a gate metric.
To verify, check that the ship rule is written down where reviewers see it, for example in the repository next to the eval code, and that the last three prompt or model changes each have an eval result showing the rule was met.
Keep capability evals and regression evals as separate suitesShould
Why it matters and how to check
Anthropic's article on agent evals distinguishes capability evals, which ask what the system can do and should start at a low pass rate, from regression evals, which check it still does what it used to and should sit near 100%. Mixing them hides both signals: a regression looks like noise inside a hard suite.
To verify, confirm the two suites are reported separately. When a capability eval passes consistently, move its tasks into the regression suite and add harder capability tasks.