AI evals cheat sheet

1. 1What evals areThe parts every eval has, and why judging changes by feel stops working after the first week.5 concepts

   1. 1EvalA repeatable test of an AI feature: a fixed set of inputs, a way to grade each output, and a score you can compare between versions.
   2. 2Dataset, task, grader, scoreThe dataset holds cases; the task is your app or prompt producing an output; the grader decides pass, fail or a score; the score is aggregated across cases.
   3. 3Evals vs unit testsUnit tests expect one exact result.
   4. 4Public benchmarks vs your evalsBenchmarks score models on shared tasks; your evals score your feature on your users' tasks.
   5. 5Eval-driven developmentWrite a handful of cases before changing the prompt, run them, make the change, run them again.

   5 in this chapterOpen this chapter
2. 2Golden datasetsWhere good cases come from, what each case needs and how to keep the set honest as it grows.8 concepts

   1. 1Golden datasetA versioned collection of inputs with the expected answer, or the rules a good answer must follow, chosen to represent real use.
   2. 2Where cases come fromReal traces and support tickets first, then bug reports, known hard cases, and cases written by domain experts.
   3. 3Case formatAn id, the input, the expected output or rubric, optional context (expected sources, allowed tools) and tags for the case type.
   4. 4CoverageCommon cases, each failure type you have seen, edge cases (empty, very long, other languages), out-of-scope requests and adversarial ones.
   5. 5Should-refuse casesInputs where the right behaviour is to say "I do not know", decline or ask a clarifying question.
   6. 6Synthetic casesCases generated by a model, for example questions written from your documents, then reviewed by a person.
   7. 7Keeping eval cases out of promptsEval cases must not appear as few-shot examples, in fine-tuning data or pasted into the system prompt.
   8. 8Versioning the setKeep the dataset in Git or a dataset tool with versions, and record which version every run used.

   8 in this chapterOpen this chapter
3. Metrics by task typePick the cheapest metric that captures what "correct" means for the task. Code checks first, judges only where needed.7 conceptsOpens with credits
4. LLM-as-judgeUsing a model to grade outputs: how to set it up, the biases it brings and how to check it before trusting it.9 conceptsOpens with credits
5. RAG metricsScore retrieval and generation separately, so you know which half to fix.7 conceptsOpens with credits
6. Agent and tool-use metricsAgents need scores for the end state, the path and the consistency across runs.6 conceptsOpens with credits
7. Offline, online and in CIWhen each kind of eval runs, what it catches and how to keep it affordable.8 conceptsOpens with credits
8. Human reviewPeople define what good means. Evals scale that judgement; they do not replace it.5 conceptsOpens with credits
9. ToolsPopular open source and hosted options, as of October 2026. Learn the method first; any of these can run it.7 conceptsOpens with credits
10. A starter eval planA first week for an app with no evals yet. Small, real and running in CI by Friday.5 conceptsOpens with credits
11. 11Glossary, A to ZEvery concept in the sheet and the smaller terms it uses, in one list.73 terms

Evals for LLM apps and agents: golden datasets, metrics, LLM-as-judge and its biases, RAG and agent metrics, offline and online evals, CI, human review, tools.