Chapter 2 of 7 · 5 practices · 3 min · 4-question quiz
Evals before shipping
Evals are automated tests for model behaviour. Without them, every prompt or model change is a guess you only check in production.
Start: Define measurable success criteria before you write the prompt5 practices in this chapter
- 1Define measurable success criteria before you write the prompt"Good answers" cannot be tested. Before you write the prompt, decide what success looks like in terms you can measure. Anthropic's eval guide recommends criteria that are specific and measurable, and…
- 2Build the eval set from real inputs, including the awkward onesSynthetic, tidy examples make every model look good. To learn how your app really behaves, test it on what real users send.
- 3Grade automatically: code checks first, then model-graded rubricsHand-grading does not scale to every change. Automate it, and choose the grader by what you are checking. Use code wherever you can. It is fast, cheap and never disagrees with itself.
- 4Run a regression eval on every prompt, model or retrieval changeChanging one sentence in a prompt can fix the case you were looking at and break five others. Only a full regression run shows both effects.
- 5Capture user feedback and link it to the logged callOffline evals miss what real users do. A lightweight feedback signal closes that gap. Pick a signal that costs the user almost nothing: thumbs up or down; an edit to the generated text; whether the…
Check yourself on this chapter
4 questions, right here once you are signed in. Your progress is saved as you read.