Chapter 5 · Practice 4 of 4
Stream interactive responses and batch offline work
From Cost and latency
Why it matters
How you call the model should depend on whether someone is waiting for the answer.
| Use streaming when | Use batch or a queue when |
|---|---|
| A user is watching, as in chat | Nobody is waiting on the result |
| Progress within a second or two matters | Jobs run nightly, such as summaries, tagging or eval runs |
| The full answer may take a while | You want controlled concurrency |
For offline work, use your provider's batch API where one exists and check its pricing, or a queue with controlled concurrency.
Measure time to first token and total time separately, because they respond to different fixes:
- shorter prompts and caching help the first;
- smaller outputs help the second.