What benchmarks can and cannot tell you

Benchmarks are fixed test sets, such as maths problems, coding tasks or exam questions, with a published score per model. They are useful for spotting big differences between generations. They have three weaknesses.

Saturation: once top models all score near the maximum, the benchmark stops separating them. Contamination: test questions leak into training data, so a high score can mean the model has seen the answers. Mismatch: the benchmark task is rarely your task, in your language, with your data.

Read benchmark scores as a rough signal of general ability, prefer recent and harder benchmarks, and never treat a one-point difference as meaningful.