The request flow

Request flow end to end

Also called GenAI architecture, LLM app architecture

What

Frontend sends the message to your backend API. The backend checks who the user is, validates input and calls the AI orchestrator, which retrieves context, calls the LLM and any tools, then streams the answer back.

Why

Drawing this path once tells you where every concern lives: auth in the API, prompts in the orchestrator, data in the stores. Most bugs sit at the joins.

browser -> Node route (auth, validate, rate limit)
  -> orchestrator (guards, memory, rewrite)
     -> retriever -> vector + keyword store
     -> LLM  <-> tools (APIs, DB, search)
  <- stream tokens back over SSE
  -> save turn, trace, cost

AI orchestrator

Also called AI service layer, pipeline

What

The code that turns one user message into model calls: it loads memory, rewrites the query, retrieves, builds the prompt, runs the tool loop and applies guards.

Why

Keeping it in its own module, apart from HTTP handlers, lets you test it, run it from evals and swap models without touching routes.

Model client wrapper

Also called LLM client, provider adapter

What

One small module that every model call goes through. It sets the model id from config, timeouts, retries and logging of tokens and cost.

Why

Changing provider or model becomes a config change, and cost tracking is impossible to forget.

export async function complete(
  messages: Msg[],
  model = env.CHAT_MODEL,
) {
  const started = performance.now();
  const resp = await client.chat({
    model, messages, timeout: 30_000,
  });
  logUsage(model, resp.usage, performance.now() - started);
  return resp;
}

Data stores in a GenAI app

What

Usually four: the main database (users, chats, messages), a vector index (often inside Postgres), object storage for uploads, and a cache such as Redis.

Why

Knowing which data lives where decides backups, deletion requests and what an attacker could reach through a tool.

Request path vs background jobs

Also called job queue, workers

What

Chat replies run in the request. Slow work such as parsing uploads, embedding documents and nightly evals runs in background workers fed by a queue.

Why

A 40-page PDF should not block a web request for a minute; the user sees "processing" and gets notified when it is ready.