Core concepts: scale, latency and availability

The vocabulary every design conversation uses. Each idea here is a trade-off you will weigh again in every later chapter.

Also called scale up, scale out

What: Vertical scaling gives one machine more CPU, memory or disk. Horizontal scaling adds more machines and spreads the work across them.
Why: Vertical scaling is simple and needs no code changes but hits a hardware ceiling and keeps a single point of failure. Horizontal scaling has no hard ceiling and adds redundancy, at the cost of distributing state and traffic.

Watch out

Scaling out a service that keeps session state in memory: requests land on a different machine and users appear logged out.

What: A stateless service keeps no per-user data between requests in its own memory. Sessions, uploads and caches live in a shared store such as a database, Redis or object storage.
Why: Any instance can serve any request, so you can add, remove or replace instances freely behind a load balancer. It is the precondition for easy horizontal scaling and autoscaling.

What: Latency is how long one request takes. Throughput is how many requests the system completes per unit of time, such as requests per second.
Why: They pull against each other: batching raises throughput but adds latency. Users feel latency, capacity planning is about throughput, so state which one a design optimises.

Also called tail latency, p99, p50, p95, percentile

What: The p99 is the latency that 99% of requests beat. The p50 is the median. Tail percentiles describe the slowest requests rather than the typical one.
Why: Averages hide the slow requests that users complain about. When one page calls many services, the chance that at least one call hits its tail grows fast, so tail latency sets the user experience.

Watch out

Averaging percentiles across servers: a mean of p99 values is not the p99 of the combined traffic.

Also called uptime, high availability, HA, five nines, nines

What: Availability is the share of time a system serves requests successfully, usually written as nines. 99.9% allows about 43 minutes of downtime a month, 99.99% about 4.3 minutes.
Why: Each extra nine costs far more in redundancy, automation and process. Agreeing the target first stops you over-building a hobby app or under-building a payments system.

Watch out

Chaining dependencies multiplies availability: two services at 99.9% in series give about 99.8% overall.

Also called SPOF, failover

What: A single point of failure is any component whose failure takes the whole system down. Redundancy removes it by running spare copies, active-active or active-passive, with failover between them.
Why: Machines, disks, zones and deployments all fail. Walking every box and arrow on a diagram and asking what happens if it dies is the quickest way to find a design weakness.

Watch out

Redundant servers that share one database, one DNS provider or one availability zone still share a single point of failure.

Also called service level objective, service level agreement, SLI, SLO, SLA, service level indicator

What: An SLI is a measured indicator, such as the share of requests under 300 ms. An SLO is the internal target for it. An SLA is a contract with customers, with penalties if missed.
Why: SLOs turn "fast and reliable" into numbers you can design, alert and trade against. The gap between 100% and the SLO is the error budget a team may spend on risky releases.
Example: SLI: successful checkout requests divided by all checkout requests. SLO: 99.95% over 28 days. SLA: 99.9% monthly, or the customer gets a 10% credit.

Go deeper