Chapter 5 · Practice 3 of 4
Set token limits and per-user quotas
From Cost and latency
Why it matters
Without limits, one user or one bug can run up the whole month's bill. Put caps at every level.
- Output: set
max_tokensor the equivalent on every call, to the lowest value that fits the task. - Input: trim history, summarise old turns and limit attachment sizes.
- Per user and per account: add rate limits and daily budgets.
Then watch spend:
- Alert when daily spend crosses a threshold.
- Break cost down by feature, so you know which part of the product is responsible.