Tauseef FayyazA to Z
Glossary
Every concept in this sheet, plus the smaller terms it uses. Select a concept to open its card.
C
- Case format
- An id, the input, the expected output or rubric, optional context (expected sources, allowed tools) and tags for the case type.
- Coverage
- Common cases, each failure type you have seen, edge cases (empty, very long, other languages), out-of-scope requests and adversarial ones.
D
- Dataset, task, grader, score
- The dataset holds cases; the task is your app or prompt producing an output; the grader decides pass, fail or a score; the score is aggregated across cases.
E
- Eval
- A repeatable test of an AI feature: a fixed set of inputs, a way to grade each output, and a score you can compare between versions.
- Eval-driven development
- Write a handful of cases before changing the prompt, run them, make the change, run them again.
- Evals vs unit tests
- Unit tests expect one exact result.
G
- Golden dataset
- A versioned collection of inputs with the expected answer, or the rules a good answer must follow, chosen to represent real use.
K
- Keeping eval cases out of prompts
- Eval cases must not appear as few-shot examples, in fine-tuning data or pasted into the system prompt.
P
- Public benchmarks vs your evals
- Benchmarks score models on shared tasks; your evals score your feature on your users' tasks.
S
- Should-refuse cases
- Inputs where the right behaviour is to say "I do not know", decline or ask a clarifying question.
- Synthetic cases
- Cases generated by a model, for example questions written from your documents, then reviewed by a person.
V
- Versioning the set
- Keep the dataset in Git or a dataset tool with versions, and record which version every run used.
W
- Where cases come from
- Real traces and support tickets first, then bug reports, known hard cases, and cases written by domain experts.