Stage 2 · Lesson 5 of 7
A frozen evaluation set
A is a fixed, versioned collection of examples with known correct answers that every candidate model or prompt is scored on. It is kept apart from training data and changed only on purpose, with a new version when it changes.
Comparing a new model with the live one is only fair on identical examples, and every quality gate later in this roadmap depends on it. Include the cases that matter to the business, such as important customer groups or hard inputs, not only a random sample, and keep a note of why each group is there. For time-based data, evaluate on a period after the training data.
Common trapletting evaluation examples leak into training, which makes every new model look better than it really is.
A question to check yourself, an AI practice session and 1 cheat sheet entry come with the full roadmap. Unlock
Resources for this stage
Start with
- CourseFreeMade With ML
Goku Mohandas · Advanced
Free: the versioning and data testing lessons, built around one real project