Chapter 2 of 8 · 5 checks · 3 min
Model choice and routing
The biggest single lever on cost per token is which model you call. These checks make sure each task uses the smallest model that passes your evals.
Use the smallest model that passes your evals for each taskMust
Why it matters and how to check
OpenAI's production guide suggests starting with the most capable model to establish what good looks like, then testing whether cheaper models give the same results with lower latency and cost. Classification, extraction, routing and short rewrites often work well on small models; open-ended reasoning may not.
To verify, for each feature keep an eval run that compares at least two model sizes on the same dataset, with quality, cost and latency side by side. The model in production should be the cheapest one that meets the feature's quality threshold.
Route requests by difficulty instead of sending everything to one modelShould
Why it matters and how to check
Most traffic is easy. A router, which can be rules, a small classifier or a small model, sends simple requests to a cheaper model and hard ones to a larger model. A cascade is a variant: try the small model first and escalate only when its answer fails a check, such as schema validation or a confidence rule. Anthropic lists making routing decisions as one use of counting tokens before a call.
To verify, log which model handled each request and measure the share of traffic going to the large model. Run your eval set through the router and confirm quality matches sending everything to the large model.
Pick a model per step in agents and pipelines, not one for the whole flowShould
Why it matters and how to check
In a multi-step flow, the planning step may need a strong model while summarising tool output, extracting fields or checking format can run on a small one. Using the large model for every step multiplies cost by the number of steps.
To verify, list the steps of each agent or pipeline with the model each uses and the share of tokens it consumes. For the two most expensive steps, run an eval with a smaller model and record the result.
Set reasoning or thinking effort deliberately where the API offers itShould
Why it matters and how to check
Many current models can spend extra tokens reasoning before they answer, and those tokens are billed. That pays off for hard problems and wastes money on simple ones. Where your provider exposes a setting for reasoning effort or a thinking budget, choose it per feature based on evals rather than leaving the highest setting everywhere.
To verify, compare output token counts and quality for each feature at two effort levels on your eval set, and record the chosen setting with the evidence in the feature's config.
Re-run model comparisons when providers release new models or change pricesNice to have
Why it matters and how to check
A smaller model released this quarter may match last year's larger one. Prices also change. A cheap, scheduled re-evaluation keeps you from paying for capacity you no longer need.
To verify, keep a calendar reminder or ticket each quarter to re-run the model comparison eval for your top three features by spend, and record the outcome even when nothing changes.