Chapter 5 · Practice 1 of 4
Choose the smallest model that passes your evals
From Cost and latency
Why it matters
Larger models are slower and cost more per token. Many tasks, such as classification, extraction and routing, work well on smaller, cheaper models.
- Pick two or three model tiers.
- Run your eval set on each and compare quality, latency and cost.
- Choose the cheapest one that meets your criteria.
Consider routing: a small model handles most requests and escalates hard ones to a larger model.
Re-run the comparison when providers release new models, because the cheapest passing option changes often.