Reasoning and quality

When thinking harder pays off: reasoning models and effort settings, why errors compound over many steps, instruction following, and the simple sum that says when a bigger model is worth it.

Reasoning model

Also called thinking model, reasoning models

What: A model that writes hidden working before its final answer. Many current models can switch reasoning on or off, or vary how much they do.
Why: Reasoning helps on maths, planning, debugging and messy documents, and adds little to simple tagging or rewriting.

Watch out

Turning reasoning on everywhere because it sounds like a quality upgrade.

Reasoning tokens

Also called thinking tokens

What: The tokens a model spends thinking before its visible answer. They are usually billed at the output price and add waiting time.
Why: They can be most of the bill on a reasoning-heavy route, so count them when you work out cost per task.

Watch out

Estimating cost from the visible answer length only.

Effort and thinking settings

Also called effort, reasoning effort, thinking level, thinking budget

What: A request setting that controls how much a model reasons: Anthropic calls it effort, OpenAI reasoning effort, Gemini a thinking setting.
Why: Which model and how much thinking are separate choices: a middle-tier model at high effort can beat a top-tier model at low effort for less money.

Watch out

Setting effort once for the whole app instead of per task route.

Compounding errors

Also called error compounding, multi-step tasks

What: In a task with many steps, per-step accuracy multiplies: 95% per step over 10 steps is about 60% overall.
Why: It explains why agents and long pipelines often need a stronger model even when single questions do not.
Example: 0.99 to the power of 20 is about 0.82; 0.95 to the power of 20 is about 0.36. Small per-step gains matter a lot on long tasks.

Instruction following

Also called prompt adherence, following rules

What: How faithfully a model obeys your format, length, language, tone and must-not rules, including many turns into a conversation.
Why: For products it often matters more than cleverness: one stray sentence before your JSON breaks the feature.

Watch out

Judging answers by how good they sound instead of checking each rule on many inputs.

Price plus cost of errors

Also called when a bigger model pays off, expected error cost

What: Total cost per request = model price + (error rate x cost of one wrong answer). Compare models on this total, not on price alone.
Why: It settles "is the bigger model worth it?" with numbers: costly errors favour the stronger model, harmless ones the cheaper.
Example: Small: 0.1 cents, 90% right; large: 1 cent, 98% right; a wrong answer costs 20 cents to fix. Totals: about 2.1 vs 1.4 cents. Large wins.

Benchmarks

Also called benchmark scores, evals leaderboard, contamination, saturation

What: Fixed public test sets with a published score per model, for coding, maths, knowledge and agent tasks.
Why: They show big generation-to-generation differences, but saturate, leak into training data and rarely match your task.

Watch out

Treating a one or two point difference as meaningful.

Go deeper