How LLM APIs work

An LLM API takes text (and sometimes images) in and returns generated text, billed by the token. These ideas shape every request you will write.

Tokens

What: Models read and write tokens: chunks of text that are often a word or part of one. In English a token averages roughly four characters, but code, numbers and other languages vary.
Why: Price, speed, context limits and output limits are all counted in tokens, so they are the unit you budget in.

Watch out

Estimating cost from word counts, or assuming two providers tokenise the same text identically.

Context window

What: The maximum number of tokens a model can take in one request, counting the system prompt, the conversation history, tool definitions, documents and the output.
Why: It limits how much history and reference material you can send; long contexts also cost more and can dilute the model’s attention.

Watch out

Resending an ever-growing chat history until requests fail or quietly get slow and expensive. Trim or summarise old turns.

The Responses API

Also called client.responses.create

What: OpenAI’s current main endpoint. You send input (a string or a list of items) plus optional instructions and tools, and get back output items and a convenient output_text.
Why: It is where new features land (built-in tools, reasoning, conversation state). The older Chat Completions API still works but new code should start here.

Watch out

Mixing Chat Completions examples (messages, choices[0]) into Responses code; the shapes differ.

Your first request

What: Create a client (it reads the API key from an environment variable) and send a prompt. The model name comes from configuration rather than the code.
Why: Keeping keys and model names out of source code lets you rotate keys and upgrade models without a code change.

Python

import os
from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY
MODEL = os.environ["MODEL"]

response = client.responses.create(
    model=MODEL,
    input="Explain HTTP caching in two sentences.",
)
print(response.output_text)

Watch out

Hard-coding the API key in source or a notebook that later gets shared or committed.

Reading token usage

What: Every response reports how many input and output tokens it used (and cached tokens, where relevant).
Why: Logging usage per request and per feature is how you find what drives cost and catch runaway prompts early.

Python

u = response.usage
print(u.input_tokens, u.output_tokens, u.total_tokens)
print(u.input_tokens_details.cached_tokens)

Choosing a model

What: Providers offer tiers: large models for hard reasoning and coding, mid-size for most product work, small fast ones for classification and high volume.
Why: Price and latency differ many times over between tiers, so matching the model to the task is the biggest cost lever you have.

Watch out

Picking a model by leaderboard rank instead of testing a few candidates on your own examples.

Go deeper