Streaming responses

A long reply can take 10 seconds or more to finish, but the first token often arrives in under a second. Streaming sends tokens to the user as they are written, usually over server-sent events (a simple way for a server to keep pushing updates to a browser).

Users read while the model writes, so the feature feels fast. It changes your code. You get a series of small chunks instead of one response, and you join the text together yourself.

Token usage arrives in the final event, and errors can arrive halfway through. Structured output is harder to use mid-stream, because half-finished JSON does not parse. Measure time to first token and total time separately, since users feel them differently.