Best Practices for Building LLM Apps

Chapter 5 · Practice 3 of 4

Set token limits and per-user quotas

From Cost and latency

Why it matters

Without limits, one user or one bug can run up the whole month's bill. Put caps at every level.

  • Output: set max_tokens or the equivalent on every call, to the lowest value that fits the task.
  • Input: trim history, summarise old turns and limit attachment sizes.
  • Per user and per account: add rate limits and daily budgets.

Then watch spend:

  • Alert when daily spend crosses a threshold.
  • Break cost down by feature, so you know which part of the product is responsible.