Chapter 5 of 7 · 4 practices · 2 min · 4-question quiz
Cost and latency
Token costs and response times decide whether a feature is viable. Most savings come from model choice, caching and limits.
Start: Choose the smallest model that passes your evals4 practices in this chapter
- 1Choose the smallest model that passes your evalsLarger models are slower and cost more per token. Many tasks, such as classification, extraction and routing, work well on smaller, cheaper models.
- 2Put stable content first so prompt caching can reuse itProviders can cache a repeated prompt prefix and bill cache hits at a fraction of the normal input price, with faster responses too. To get those hits, you have to order the prompt for it.
- 3Set token limits and per-user quotasWithout limits, one user or one bug can run up the whole month's bill. Put caps at every level. Output: set
max_tokensor the equivalent on every call, to the lowest value that fits the task… - 4Stream interactive responses and batch offline workHow you call the model should depend on whether someone is waiting for the answer. For offline work, use your provider's batch API where one exists and check its pricing, or a queue with controlled…
Check yourself on this chapter
4 questions, right here once you are signed in. Your progress is saved as you read.