Load balancing, proxies, CDNs and caching

The components that sit in front of every system and keep repeated work off the servers and databases behind them.

Load balancer (L4 vs L7)

Also called LB, ALB, NLB

What: A load balancer spreads incoming requests across healthy servers. Layer 4 balancers route TCP or UDP connections by address and port. Layer 7 balancers read HTTP and can route by path, header or cookie.
Why: It lets you scale horizontally, removes failed servers using health checks and gives one stable entry point. Pick L4 for raw speed and non-HTTP traffic, L7 for smart routing and TLS termination.

Watch out

Forgetting the load balancer itself needs redundancy, or health checks that pass while the app cannot reach its database.

Load balancing algorithms

Also called round robin, least connections

What: Round robin sends requests to servers in turn. Least connections picks the server with the fewest active requests. Hashing on a key, such as user ID, sends the same client to the same server.
Why: Round robin suits similar, short requests. Least connections suits uneven request cost. Hashing gives stickiness for caches or sessions, but uneven keys can overload one server.

Reverse proxy and API gateway

Also called reverse proxy, API gateway

What: A reverse proxy receives client requests and forwards them to backend servers, handling TLS, compression and caching. An API gateway is a reverse proxy for APIs that adds authentication, rate limiting, routing and request shaping.
Why: They centralise cross-cutting concerns so each service does not reimplement them, and hide the internal layout of services from clients.

Watch out

Putting business logic in the gateway, which turns it into a hard-to-change bottleneck that every team must edit.

CDN

Also called content delivery network, edge cache

What: A content delivery network is a set of servers around the world that cache content close to users. Pull CDNs fetch from your origin on the first request; push CDNs are uploaded to ahead of time.
Why: It cuts latency for static files, images and video, and absorbs traffic spikes and some attacks before they reach your origin servers.

Watch out

Caching personalised responses at the edge, so one user sees another user's page. Set Cache-Control: private on them.

Cache-aside

Also called lazy loading, look-aside cache

What: The application checks the cache first. On a miss it reads the database, stores the result in the cache with a TTL, and returns it. Writes go to the database and delete or overwrite the cache entry.
Why: It is the most common caching pattern for read-heavy data: only requested data is cached, and a cache outage slows the system rather than breaking it.

Python

def get_user(user_id):
    key = f"user:{user_id}"
    if (cached := cache.get(key)) is not None:
        return cached
    user = db.fetch_user(user_id)
    cache.set(key, user, ttl=300)
    return user

Watch out

Updating the cache before the database commits, which can leave stale data in the cache for the whole TTL.

Cache invalidation and TTL

Also called time to live, stale data, TTL, cache invalidation

What: Invalidation removes or refreshes cached data when the source changes. A TTL is an expiry time after which an entry is discarded, which bounds how stale it can be.
Why: Every cache trades freshness for speed. Choosing a TTL per data type, and deleting keys on writes where freshness matters, decides what users may see out of date and for how long.

Watch out

No TTL at all: one missed invalidation leaves wrong data cached forever.

Cache eviction policies

Also called LRU, LFU, cache eviction policy, eviction policy

What: When a cache is full, an eviction policy decides what to remove. LRU removes the least recently used entry, LFU the least frequently used, and TTL-based eviction removes expired entries first.
Why: Memory is limited and expensive. The right policy keeps the hot working set in memory, so the hit ratio stays high as data grows.

Cache stampede

Also called thundering herd, dogpile

What: A cache stampede happens when a popular key expires and many requests miss at once, all hitting the database to rebuild the same value.
Why: It can take a database down exactly when traffic peaks. Fixes include request coalescing (one rebuild, others wait), a lock per key, refreshing early and adding random jitter to TTLs.

Go deeper