Latency, throughput and percentiles

Latency is how long one request takes. Throughput is how many requests the system finishes each second. They are linked but not the same.

Batching, handling many items in one go, raises throughput but makes each item wait longer. A queue that is nearly full keeps throughput high while latency climbs sharply. Measure latency as percentiles, not averages.

The p50 is the time half of all requests beat; the p99 is the time 99 in every 100 beat. If p50 is 40 ms and p99 is 900 ms, one request in a hundred is slow. A page that makes 20 backend calls then hits that slow path on roughly one load in five.

Learn the rough cost of common operations too. Reading from memory takes nanoseconds. A round trip inside the same datacentre takes about half a millisecond. A round trip across continents takes 100 ms or more.