Sixteen Things That Break a Backend, and What Prevents Each

The part that is not the code

Backend engineering is taught as frameworks and taken as frameworks, and then the actual job turns out to be almost entirely about the things around the code: what happens when the database is slow, when the same request arrives twice, when a downstream service stops answering but does not have the courtesy to say so.

None of that is difficult in the sense of being intellectually deep. It is difficult in the sense that it is invisible until it happens to you, usually at an unpleasant hour, and the fix is obvious in retrospect every single time.

So here are sixteen of those situations with their usual answers. Read them as defaults for a system that has real users, because almost all of them are unnecessary until suddenly they are not optional.

The database is the hard part

1. Data + more than one reader → a real database. Files and in memory structures are fine until two processes touch them, at which point you are writing a database badly. Reach for the boring relational one first, and require a specific reason to use anything else.

2. A query + a table that has grown → an index. Read the query plan rather than guessing, because the index you assumed was being used frequently is not. Understand also what you are buying it with, which is slower writes and more storage, so an index on everything is its own kind of mistake.

3. Two writes that must both happen → a transaction. Money moving, a record and its children, anything where half the operation is worse than none of it. Know what your isolation level actually guarantees, because the default in most databases is weaker than most developers assume.

4. A schema change + live traffic → expand, migrate, contract. Add the new column, write to both, backfill, move reads across, then remove the old one. Every step is reversible. The single deploy that renames a column is how a routine migration becomes an outage.

Serving one request

5. A request + work that can wait → a background job. Emails, thumbnails, exports, anything calling a third party. The user is holding a connection open while you do it, and every second there is a second where a timeout somewhere can lose work you have already half done.

6. A retry + something that costs money → an idempotency key. Networks lose responses, not just requests, so the client cannot tell a failed charge from a successful one it never heard about. Let the caller supply a key and return the original result for a repeat, and double charging stops being possible.

7. A list endpoint + a growing table → pagination from day one. Every unbounded list is a future incident with a known date. Prefer cursors over offsets once the data is large, because offset pagination gets slower the deeper anyone goes and quietly skips rows when things are inserted mid scan.

8. Expensive reads + repetition → a cache, and a plan for invalidating it. The cache is the easy half. Decide before you add it how an entry becomes wrong and what happens then, because a cache without an invalidation story does not fail loudly, it just serves the wrong answer to a fraction of people indefinitely.

Failing well

9. Any network call + no timeout → set one. Most default clients wait effectively forever. One slow dependency then consumes every worker you have, and a service that was merely degraded takes down a service that was completely healthy. This one line is the highest return resilience change available.

10. A failure + a retry → back off, add jitter, cap the attempts. Immediate uniform retries are how a brief blip becomes a sustained outage, because every client returns at the same instant with the same enthusiasm. Retry only what is safe to repeat, which loops you back to heuristic six.

11. Traffic spikes + a shared resource → rate limits and backpressure. Deciding what to shed is a design decision you should make deliberately, in advance, while calm. If you do not, the system will make it for you at random, and random is the worst possible policy for choosing which users to disappoint.

12. An incident + a question about what happened → a request id in structured logs. One identifier flowing through every service and every log line, so a single user complaint can be traced end to end. Add it before you need it, because during an incident is when you will most regret not having it.

The mistakes that are expensive to undo

13. A secret + a repository → environment variables, and rotate anything committed. Assume anything that reached a commit is public forever, because history is public even after the file is deleted. Rotate first, then clean the history, never the other way round.

14. A password + storage → a slow hash, salted, per user. Argon2 or bcrypt, never a general purpose hash, never anything reversible. This is one of the few areas of engineering where inventing your own approach is not merely unwise but close to negligent, and the correct implementation is one library call.

15. A backup + a restore nobody has attempted → perform the restore. A backup you have never restored is a belief rather than a backup, and the moment of discovery is always the worst available moment. Restore it into a scratch database once, write down how long it actually took, and put the next attempt in the calendar.

16. A scaling problem + an instinct → find the bottleneck first. Adding servers to a system limited by a single database connection pool achieves nothing but a larger bill. Measure, find the actual constraint, fix that one thing, then measure again, because the bottleneck moves every time you relieve it.

What else belongs on this list

Events instead of chained synchronous calls, once you have more than a couple of services, because a chain of five calls has the availability of the worst link and the latency of all of them. Health checks that check something real rather than returning two hundred unconditionally. And connection pool limits, which turn out to be the binding constraint far more often than anyone expects and are almost never set deliberately.

The one I would put above all of those is knowing what your system is allowed to lose. Some data can be regenerated, some can be lost for a minute, and some can never be lost at all. Teams that have never had that conversation end up protecting all three equally, which means paying too much for the cheap data and not enough for the expensive kind.

The short version

Real database, correct indexes, transactions where they matter, migrations that expand before they contract. Move slow work off the request. Make retries safe with idempotency keys. Paginate everything. Cache with an invalidation plan. Timeouts on every call, backoff with jitter on every retry, rate limits before you need them, a request id in every log line. Secrets out of the repository, passwords through a slow hash, a restore you have actually performed. Measure before you scale.

Nothing on that list is clever, and that is the point. Backend engineering rewards the boring correct move performed consistently far more than it rewards anything ingenious, because the systems that stay up are the ones where the ordinary failure modes were handled ordinarily.