Timeouts, retries and backoff

Treat a model API like any unreliable service across the network. Calls can hang. They can hit rate limits (HTTP 429).

They can fail with server errors or overload responses, the codes in the 5xx range. Set a timeout on every call, long enough for your longest expected reply. Retry only errors that might succeed a second time: 429 and 5xx.

Never retry a 400, which means your request itself was wrong. Wait longer before each new attempt (exponential backoff), add a little random jitter so many clients do not retry at the same instant, and honour any retry-after header. Cap the attempts, usually at three or four, then fail clearly or fall back.

Most official SDKs already retry some errors by default. Learn what yours does before adding your own retries on top, or you multiply the load.