Retry transient errors with backoff and a timeout

Rate limits and server errors are normal at scale. Retry the ones that can succeed, and fail fast on the ones that cannot.

When you retry:

• use exponential backoff with jitter;
• respect any retry-after header;
• cap the number of attempts.

Set a timeout on every call, so a stuck request does not hold a user or a worker forever.

for (let attempt = 0; attempt < 4; attempt++) {
  try {
    return await callModel(req, { signal: AbortSignal.timeout(30_000) });
  } catch (err) {
    if (!isRetryable(err) || attempt === 3) throw err;
    await sleep(2 ** attempt * 500 + Math.random() * 250);
  }
}