Checklist · AI
LLM Evaluation Checklist
Opens with creditsA checklist for evaluating LLM features and agents before and after launch. It covers success criteria, building the eval dataset, choosing and calibrating graders, running evals reproducibly, agent and multi-step evals, safety and security tests, and production monitoring, with a severity and a concrete verification step for each check.
Written by Tauseef Fayyaz7 chapters, 32 checksAbout 14 minReviewed 13 October 2026What you will learn
7 chapters · about 14 min
1Success criteria and scopeAn eval measures progress toward a goal, so the goal comes first. These checks make sure everyone agrees what good looks like before anything is graded.4 checks · 2 min
2Eval datasetThe dataset decides what your eval can see. These checks make it representative, unambiguous and safe to keep.5 checks · 2 min
- 1Build the dataset from real or realistic inputs, not tidy examples
- 2Include edge cases on purpose: empty, long, ambiguous, off-topic and hostile inputs
- 3Make each case unambiguous, with a reference answer or grading rule
- 4Start with a small set from real failures and grow it with every bug
- 5Version the dataset and keep a held-out slice you never tune against
5 in this chapterOpen this chapter- Graders and LLM judgesA grader turns an output into a score. These checks pick the cheapest reliable grader for each criterion and make sure model judges agree with people.5 checks · 2 minOpens with credits
- Running evals reproduciblyAn eval score is only useful if you can reproduce it and compare it with the last one. These checks make runs consistent and part of every change.5 checks · 2 minOpens with credits
- Agents and multi-step systemsAgents and pipelines can fail at any step and still reach a plausible answer. These checks evaluate the steps and the final state, not only the text at the end.5 checks · 2 minOpens with credits
- Safety, security and privacy evalsAccuracy evals do not show whether a feature can be manipulated or can cause harm. These checks add adversarial and safety cases to the same pipeline.4 checks · 2 minOpens with credits
- Production evaluation and monitoringOffline evals predict behaviour; production shows it. These checks connect live traffic back to the eval suite.4 checks · 2 minOpens with credits
About this checklist
Most LLM features are tested the same way at first: someone tries a handful of prompts, the answers look good, and the feature ships. OpenAI's evaluation guide has a name for this, vibe-based evaluation, and lists it as an anti-pattern. This LLM evaluation checklist is the alternative: a set of checks that turn "it seems fine" into numbers you can track across every prompt, model and retrieval change.
The checklist follows the published evaluation guidance from Anthropic and OpenAI, Anthropic's engineering write-up on evaluating agents, research on using language models as judges, and NIST's Generative AI Profile for risk measurement. It applies whether you evaluate a single prompt, a retrieval pipeline or an agent that runs for dozens of steps.
The chapters move from defining success, through building the dataset and choosing graders, to running evals reproducibly and keeping them alive in production. Agents and safety each get a chapter, because both need tests that ordinary accuracy metrics miss: whether the agent left the world in the right state, and whether the feature can be talked into doing harm.
Each check has a severity (must, should or nice) and a concrete way to verify it, such as a file to look for, a number to compute or a test to run. If you are starting from nothing, do the must checks in the first three chapters first. They alone will catch most regressions before your users do.
Common questions
What is an LLM eval?
An LLM eval is an automated test that runs a model, prompt or agent over a fixed set of inputs and grades the outputs against success criteria, using code checks, a model judge or human review. It lets you change a prompt or model and see whether quality went up or down before users do.
Can I use an LLM as a judge?
Yes, for subjective qualities such as tone or helpfulness, but calibrate it first. Give it a clear rubric and a constrained answer, check its agreement with human labels on a sample, and control for known biases such as position and verbosity. Use code-based graders for anything rules can check.
How many test cases do I need to start evaluating an LLM app?
Fewer than you might think. Anthropic suggests 20 to 50 simple tasks drawn from real failures as a good start for agents. Grow the set with every bug you fix, and once grading is automated, add volume, since many slightly noisy cases beat a few hand-graded ones.
What is the difference between pass@k and pass^k?
pass@k is the chance that at least one of k trials succeeds, which rises as k grows. pass^k is the chance that all k trials succeed, which falls as k grows. Use pass^k for features users rely on every time, since a 75% per-trial rate gives only about 42% over three trials.
How do I evaluate an AI agent differently from a single prompt?
Grade the final state of the environment rather than only the final message, allow different valid paths, run several isolated trials per task, and measure tool selection, argument accuracy and handoffs. Read transcripts regularly to separate agent failures from grader bugs.
Sources (8)
Checked against these, last on 13 October 2026.
- Define success criteria and build evaluations, Anthropic docsplatform.claude.com/docs/en/test-and-evaluate/develop-tests
- Demystifying evals for AI agents, Anthropic Engineeringwww.anthropic.com/engineering/demystifying-evals-for-ai-agents
- Evaluation best practices, OpenAI docsdevelopers.openai.com/api/docs/guides/evaluation-best-practices
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, arXivarxiv.org/abs/2306.05685
- OWASP Top 10 for LLM Applications 2026, OWASP GenAI Security Projectgenai.owasp.org/resource/owasp-genai-llm-top-10-2026/
- Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1), NISTwww.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence
- Inspect, UK AI Security Instituteinspect.aisi.org.uk/
- Writing effective tools for agents, Anthropic Engineeringwww.anthropic.com/engineering/writing-tools-for-agents
Related on Sefism
Picked from the other libraries by shared topics.
Roadmaps
See allCheat sheets
See all- AI evals cheat sheetEvals for LLM apps and agents: golden datasets, metrics, LLM-as-judge and its biases, RAG and agent metrics, offline and online evals, CI, human review, tools.
- Coding with AI tools cheat sheetClaude Code, Cursor, Copilot, used well.
- AI engineering cheat sheetBuild products on LLMs and generative AI.
Best practices
See all- Best Practices for Building LLM AppsHow to build LLM apps and agents that hold up in production: prompts, evals, prompt injection, tool permissions, cost, logging and privacy.
- Prompt Engineering Best PracticesHow to write prompts that work the same way on the hundredth input as on the first: clear instructions, structure, examples, reasoning, output control and testing.
- Best Practices for AI Coding AssistantsHow to get reliable code from AI coding assistants and agents: instruction files, task prompts, verification, review, security, dependencies and session hygiene.
Checklists
See all- RAG Production ChecklistA RAG production checklist: chunking, embeddings, hybrid search, reranking, per-user permissions, citations, evals, re-indexing and cost, with how to verify each check.
- AI Feature Launch ChecklistAn AI feature launch checklist: use case fit, evals, UX for errors, disclosure, human override, abuse filters, rate limits, cost caps, prompt privacy, monitoring and rollback.
- EU AI Act Readiness ChecklistAn EU AI Act readiness checklist for teams building or deploying AI: scope and roles, prohibited practices, risk classification, transparency, GPAI duties, oversight and dates.
Project structures
See all- GenAI RAG app (FastAPI) project structureFor developers turning a "chat with your documents" demo into an app people can rely on. A FastAPI backend where every layer has a home: ingestion workers, retrieval, prompts as versioned files, model clients with fallbacks and cost caps, guards, evals that gate CI, and tracing, so a wrong answer can be traced to the step that caused it.
- Python AI agent with LangGraph project structureFor Python developers moving from "call a model in a loop" to an agent that can pause, resume and be debugged. It starts from the official LangGraph template (langgraph.json, a src/agent package, unit and integration tests) and grows into a customer support agent with typed state, one file per node and tool, human approval before refunds, evals that score whole trajectories, tracing, auth and a deployable server image.
- AI agent with an MCP server project structureFor developers who want to give AI assistants safe access to their own data or API. A TypeScript MCP server built on the official SDK, organised by feature (tasks, projects) with each feature owning its tools, resources and prompts, served over stdio for local hosts and Streamable HTTP with OAuth for remote ones, plus a small agent and the evals that prove a model can use it.
Career questions
See all- Which AI tools do employers expect junior developers to know in 2026?Employers rarely require a specific brand. They expect you to use an AI coding assistant well (in the editor and in the terminal), to review and test what it produces, to call an LLM API in a small feature, and to explain when you would not use AI. Pick one assistant, learn it properly, and show judgement in your projects.
- How do I know if code written by an AI tool is actually correct?You check it the same way you check any code, but more carefully: read the whole diff, confirm every function and package exists, run it against tests that could fail, especially one you wrote yourself, try the edge cases, and make sure you can explain every line. Correct means proven by evidence, not "the tool said it works".
- How do I use AI to become a better engineer instead of becoming dependent on it?Decide the job AI does before you open it. Use it as a tutor, reviewer and sparring partner that explains and challenges, not a vending machine that hands you finished code. Attempt first, ask second, and always be able to explain and defend what you ship.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
- 1:1 Mentorship
- Fresh Graduate Mentorship
- Career Guidance
Brand deals, promos and partnerships: tauseeffayyaz@gmail.com