How LLMs work: tokens and context

What a model does when it answers: text becomes tokens, the model predicts one token at a time inside a fixed context window, and every token costs money and time.

Large language model

Also called LLM, foundation model

What: A neural network, almost always a transformer, trained on huge amounts of text to predict the next token. Further training on instructions and human or AI feedback turns it into an assistant that follows requests.
Why: It is the engine behind every feature in this track. Remembering that it generates likely continuations rather than looking up facts explains most of its strengths and failures.

Tokens and tokenisation

Also called tokenizer, BPE, count tokens

What: Models read and write tokens: chunks of text from a fixed vocabulary, often word pieces. In English one token is roughly four characters; code, numbers and many other languages take more tokens per word.
Why: Context limits, prices, rate limits and latency are all measured in tokens, so counting them is how you estimate cost and decide what fits in a request.

Python

count = client.messages.count_tokens(
    model="claude-sonnet-5",
    system=SYSTEM_PROMPT,
    messages=[{"role": "user", "content": document}],
)
print(count.input_tokens)

Watch out

Estimating with another vendor's tokenizer: each model family splits text differently, so use the provider's token counting endpoint.

The context window

Also called context length

What: The maximum number of tokens a model handles in one request, counting the system prompt, history, tool definitions, retrieved documents and the output. Current frontier models offer roughly 200K to 1M tokens.
Why: It is a hard limit and a quality lever: everything the model knows about the task must fit, and filling it with noise dilutes attention, slows responses and costs more.

Watch out

Assuming a huge window is used evenly: recall of details buried in long inputs drops, so trim aggressively and put what matters where it is easy to find.

Token pricing

Also called cost per token, cost per request

What: Providers charge per million input tokens and per million output tokens, with output several times more expensive. Cached input tokens and batch requests are billed at a discount.
Why: Cost per request is input tokens times the input price plus output tokens times the output price, which is how you budget a feature before building it.
Example: A call with 2,000 input and 500 output tokens on a model priced at $3 and $15 per million costs $0.006 + $0.0075 = $0.0135, so 1,000 such calls cost about $13.50.

Watch out

Forgetting that history, tool definitions and reasoning tokens are billed on every call, not only the new message.

Rate limits

Also called RPM, TPM, 429, requests per minute, tokens per minute

What: Caps on requests per minute and on input and output tokens per minute for an account or workspace. Going over returns HTTP 429, usually with a retry-after header.
Why: They cap production throughput, so you size traffic, queue background work, spread load across models and plan fallbacks around them.

Watch out

Load testing with one user and short prompts: limits bite under concurrent traffic with real context sizes.

Temperature and sampling

Also called temperature, top-p, top-k

What: The model outputs a probability for every possible next token and sampling picks one. Low temperature sharpens the distribution, high temperature flattens it; top-p and top-k limit choices to the most likely tokens.
Why: Low randomness suits extraction and classification, higher suits brainstorming. Many current reasoning models fix sampling and reject these parameters, so you steer them with prompts and effort instead.

Watch out

Believing temperature 0 makes output deterministic or correct: it only reduces variation, and server-side batching can still change results.

Hallucination

Also called confabulation, why models hallucinate

What: A fluent, confident answer that is false or unsupported, such as an invented citation, API, number or policy. It happens because the model produces plausible text, not verified facts.
Why: It is the main quality risk in LLM products, which is why you ground answers in retrieved sources, allow the model to say it does not know, and measure factuality with evals.

Watch out

Expecting a bigger model or a "do not make things up" instruction to fix it: grounding, citations and automated checks do far more.

Reasoning models and effort

Also called thinking, extended thinking, adaptive thinking, chain of thought

What: Models that generate internal reasoning tokens before the final answer. APIs expose a setting such as effort or thinking that trades depth of reasoning against tokens and latency.
Why: Multi-step analysis, coding and agent planning gain a lot from reasoning while simple classification does not, so you tune effort per route instead of globally.

Watch out

Leaving maximum effort on for high-volume simple calls: reasoning tokens are billed as output and add seconds of latency.

Choosing a model

Also called model selection

What: Picking a model by measuring quality on your own eval set against cost and latency. Providers ship tiers from most capable to fastest and cheapest, such as Claude Opus, Sonnet and Haiku.
Why: The right model is usually the cheapest one that passes your evals for that task, and different routes in one product often use different models.

Watch out

Choosing from public leaderboards alone: they rarely match your data, prompts or failure modes.

Go deeper