Chapter 1 of 8 · 4 checks · 2 min
Measuring cost
You cannot cut what you cannot see. These checks give you cost per request, per feature and per user, from the usage data every API already returns.
Log the full token usage of every call, including cached and output tokensMust
Why it matters and how to check
Every response carries a usage object. On Anthropic's API, total input is
cache_read_input_tokens + cache_creation_input_tokens + input_tokens, andinput_tokensalone is only the part after the last cache breakpoint, so logging just that field undercounts. On OpenAI's API, cached input appears inusage.input_tokens_details.cached_tokens. Log all input fields, output tokens, the model and a request id for every call.To verify, pick ten logged calls and recompute their cost from the logged fields and your provider's current price list. The total should match the provider's usage dashboard for the same calls within rounding.
Tag every call with the feature, prompt version and user or tenantMust
Why it matters and how to check
A single monthly total tells you nothing about what to fix. Tag each call with the product feature, the prompt or agent version, the environment and the user or tenant, so you can see that, for example, one summarisation job is most of the bill, or one customer's usage doubled overnight.
To verify, build a report of last week's spend grouped by feature and by top ten users or tenants. If more than a small share of spend falls under "unknown", find the untagged call sites and fix them.
Track cost per successful task, not only cost per callShould
Why it matters and how to check
An agent that takes twelve calls to finish a task, or a feature that users retry three times because the first answer was poor, costs more than its per-call price suggests. Divide spend by the number of tasks that actually succeeded, using your eval or product success signal.
To verify, add a cost per successful task metric to the dashboard for each feature, and check it next to quality in every eval run, so a change that cuts per-call cost but lowers success rate is visible.
Count tokens before shipping a prompt or model change, on the target modelShould
Why it matters and how to check
Token counts depend on the tokenizer, which can differ between model families from the same provider. Anthropic's token counting docs note that its newer tokenizer produces roughly 30 percent more tokens for the same text than earlier models, and advise recounting prompts against the model you plan to use. Anthropic's token counting endpoint is free to use, within its own rate limits.
To verify, for any model migration, count a sample of your real prompts on the old and new model and record the difference in the change description, alongside the price difference, before approving the switch.