LLM Evaluation Checklist

What this checklist is

A checklist for evaluating LLM features and agents before and after launch. It covers success criteria, building the eval dataset, choosing and calibrating graders, running evals reproducibly, agent and multi-step evals, safety and security tests, and production monitoring, with a severity and a concrete verification step for each check.

It has 7 chapters and 32 checks, each with how to verify it and how much it matters. For developers testing LLM features and agents.

Chapter by chapter

1. Success criteria and scope (4 checks)

An eval measures progress toward a goal, so the goal comes first. These checks make sure everyone agrees what good looks like before anything is graded.

2. Eval dataset (5 checks)

The dataset decides what your eval can see. These checks make it representative, unambiguous and safe to keep.

3. Graders and LLM judges (5 checks)

A grader turns an output into a score. These checks pick the cheapest reliable grader for each criterion and make sure model judges agree with people.

4. Running evals reproducibly (5 checks)

An eval score is only useful if you can reproduce it and compare it with the last one. These checks make runs consistent and part of every change.

5. Agents and multi-step systems (5 checks)

Agents and pipelines can fail at any step and still reach a plausible answer. These checks evaluate the steps and the final state, not only the text at the end.

6. Safety, security and privacy evals (4 checks)

Accuracy evals do not show whether a feature can be manipulated or can cause harm. These checks add adversarial and safety cases to the same pipeline.

7. Production evaluation and monitoring (4 checks)

Offline evals predict behaviour; production shows it. These checks connect live traffic back to the eval suite.

How it works on Sefism

The checklist is one page: each check has a severity (must, should or nice to have) and a concrete way to verify it, and members can tick checks off as they go. The opening chapters are open to everyone. Adding it to your account uses credits, opens every chapter and keeps your progress.

Every claim is checked against the sources listed on the page (8 cited).