Chapter 2 · Practice 3 of 5
Grade automatically: code checks first, then model-graded rubrics
Why it matters
Hand-grading does not scale to every change. Automate it, and choose the grader by what you are checking.
| Use code checks for | Use a model grader for |
|---|---|
| Exact match on labels | Tone |
| Schema validation | Helpfulness |
| Regex for required fields | Other subjective qualities |
| Checking that cited ids exist |
Use code wherever you can. It is fast, cheap and never disagrees with itself.
For subjective qualities, use a model as a grader with a clear rubric and a constrained answer, such as a single score. Anthropic's guide recommends:
- prioritising volume over perfection, since many roughly graded cases beat a few perfect ones;
- grading with a different model from the one that produced the output, where possible.
Spot-check a sample of model-graded results by hand to confirm the grader agrees with people.
Example
function gradeLabel(output: string, expected: string) {
return output.trim().toLowerCase() === expected.toLowerCase();
}
const accuracy = cases.filter((c) => gradeLabel(c.output, c.expected)).length / cases.length;