Sefism AI Evals Cheat Sheet

What this cheat sheet is

The Sefism AI evals cheat sheet is a quick-revision book on measuring LLM features instead of guessing. It covers what an eval is, how to build a golden dataset, which metric fits which task, how to trust an LLM judge, how to score RAG answers and agent runs, how to run evals offline, online and in CI, where humans fit, and a starter plan you can follow in your first week.

It is a standalone sheet: it does not belong to one roadmap and has no editions, so every reader sees the same ten chapters and 67 concepts. Each concept is a short card with what it is, why it matters and, for many, the mistake people make with it. The AI engineering, AI agents and MCP and MLOps roadmaps teach the method stage by stage; this sheet adds the metric definitions, judge biases, agreement checks, reliability maths and tools in one place to keep open while you build. Tool facts were checked on the tools' official pages in October 2026.

Who it suits

• Developers shipping a chatbot, RAG assistant or agent who change prompts and models and want to know whether it got better.
• Students whose final year project uses an LLM and needs evidence that it works, not a demo that happened to go well.
• Anyone moving into AI engineering, where writing evals is now an everyday part of the job.

Chapter by chapter

1. What evals are

An eval as a repeatable test of an AI feature, its four parts (dataset, task, grader, score), how evals differ from unit tests, why public benchmarks do not replace your own evals, and eval-driven development.

2. Golden datasets

What a golden dataset is, where good cases come from, the case format, coverage, should-refuse cases, synthetic cases, keeping eval cases out of prompts and versioning the set as it grows.

3. Metrics by task type

Pick the cheapest metric that captures what correct means: exact match; precision, recall and F1; field-level accuracy; code assertions; rubric scoring; BLEU, ROUGE and similarity scores; and cost and latency as metrics. Code checks first, judges only where needed.

4. LLM-as-judge

Using a model to grade outputs: one criterion per judge, pairwise comparison, the biases (position, verbosity, self-preference and leniency), checking a judge against your own labels before trusting it, and pinning the judge's model and prompt.

5. RAG metrics

Score retrieval and generation separately: recall at k, mean reciprocal rank, faithfulness, answer relevance, context precision and context recall, citation correctness and abstention rate.

6. Agent and tool-use metrics

Tool-call accuracy, task completion, trajectory checks, pass@k against pass^k for consistency across runs, steps, tokens and cost per task, and fake tools for safe evals.

7. Offline, online and in CI

Offline and online evals, user signals, A/B tests, a regression gate in CI, run-to-run noise, fast and full suites, and comparing runs.

8. Human review

Error analysis, annotation guidelines, inter-annotator agreement, review queues and domain experts. People define what good means; evals scale that judgement rather than replacing it.

9. Tools

Popular open source and hosted options as of October 2026: Ragas, DeepEval, promptfoo, Langfuse, OpenAI Evals, Inspect, and a plain script. The chapter's advice is to learn the method first, because any of these can run it.

10. A starter eval plan

A first week for an app with no evals yet: collect and read real outputs, group and count the failures, build the first 30 cases, add assertions and then one judge, and wire it into CI by the fifth day.

Rules of thumb

The sheet ends with eight rules, including:

• Read real outputs before you choose a single metric.
• Prefer code assertions; use a judge only for what code cannot check.
• Make judges binary and narrow, and check them against your own labels.
• Run agent tasks several times and report consistency, not one lucky pass.
• Set CI thresholds wider than the run-to-run noise you measured.
• Turn every bug report into a new eval case the same day.

How to use it

• Before building anything, read chapters 1 and 2, and follow chapter 10 as your first week.
• When choosing how to score, use chapter 3 for the metric and chapters 5 or 6 for RAG and agents.
• Before trusting an LLM judge, work through chapter 4 and compare it with your own labels.
• When wiring evals into a pipeline, keep chapter 7 open, especially the parts on noise and thresholds.

How it works on Sefism

Everything on Sefism is free to learn from. The public page lists all ten chapters, with the first chapters open to read. Keeping the sheet in your member area uses credits, like any cheat sheet; because it is standalone, it is not included with a roadmap. New members get starting credits when they join. If you are stuck on how to evaluate your own app, send Tauseef a query.

Related on Sefism

• Roadmaps: AI engineering, AI agents and MCP, MLOps and Choosing an LLM and AI stack
• Cheat sheets: GenAI app architecture and LLM APIs
• Question: How do I prove my AI agent project actually works before I show it to anyone?
• FYP ideas: Evaluation harness for language model applications and Fine-tuned small open model with eval-gated releases
• Resources: AI evals: everything you need to know, promptfoo, DeepEval and Langfuse documentation

FAQ

What is an eval for an LLM app? A repeatable test of an AI feature: a fixed set of inputs, a way to grade each output, and a score you can compare between versions. It turns every prompt or model change into an experiment with a before and after.

When should I use an LLM as a judge? Only for what code cannot check, such as tone or whether an answer follows a rubric. Make each judge answer one narrow question, and check its verdicts against labels you wrote yourself before you rely on it.

How many eval cases do I need to start? The starter plan builds the first 30 cases from real outputs on day three, then grows the set by turning every bug report into a new case.

What is the difference between pass@k and pass^k? pass@k asks whether at least one of k runs succeeded; pass^k asks whether all k did. For agents users rely on, consistency across runs matters more than one lucky pass.