Sixteen Things That Break a Backend, and What Prevents Each

The short answer: backend development best practices are mostly about failure

Most backend development best practices have very little to do with frameworks. They are about what happens around your code: the database that gets slow, the request that arrives twice, the downstream service that stops answering without the courtesy of an error. None of it is intellectually deep. All of it is invisible until it happens to you, usually at an unpleasant hour, and the fix always looks obvious afterwards.

I am a Lead Full Stack Engineer, and almost every backend incident I have been close to traces back to one of a small set of ordinary mistakes. Students usually learn Express, Django, Spring or Laravel and assume the job is writing endpoints. Then their first production system teaches them that the endpoint was the easy part.

So here are sixteen things that break a backend, grouped by where they bite, with the default that prevents each. Treat them as defaults for any system with real users. Almost all of them feel unnecessary right up until they are not optional.

If you are earlier in your journey, the backend roadmap puts the underlying skills in order, and the backend cheat sheet is the one-page version.

Backend best practices for the database layer

The database is where most real backend difficulty lives, because it is the one component that has to be right.

1. Data and more than one reader: use a real database

Files and in-memory structures are fine until two processes touch them at once, at which point you are writing a database badly. Reach for a boring relational database first, and require a specific reason before choosing anything else. PostgreSQL or MySQL will carry a surprising amount of load before you need to think harder. If relational modelling still feels shaky, Grokking Relational Database Design and Modeling covers schemas, keys and normalisation properly, and the free PostgreSQL tutorial gets you productive quickly.

2. A query and a table that has grown: add the right index

Read the query plan rather than guessing. The index you assumed was being used frequently is not, because the query filters on a different column order or wraps the column in a function. Understand what an index costs as well as what it buys: every index slows writes and takes storage, so indexing every column is its own mistake. EXPLAIN ANALYZE in PostgreSQL is the single most useful habit here.

3. Two writes that must both happen: wrap them in a transaction

Money moving between accounts, an order and its line items, anything where half the operation is worse than none of it. Then learn what your isolation level actually guarantees. The PostgreSQL documentation states plainly that Read Committed is the default isolation level, which still allows two concurrent transactions to read the same row and both act on a value that is about to change. Most developers assume the default is stronger than it is. Grokking Database Fundamentals for Tech Interviews is a good structured way to get transactions, isolation and locking straight.

4. A schema change and live traffic: expand, migrate, contract

Add the new column, write to both old and new, backfill, move reads across, then remove the old column. Martin Fowler calls this parallel change, and every step is reversible. The single deploy that renames a column is how a routine migration becomes an outage, because the old version of your code is still running on some servers while the new schema lands.

Backend reliability when serving a request

5. A request and work that can wait: move it to a background job

Emails, thumbnails, PDF exports, anything that calls a third party. While you do that work inline, the user is holding a connection open, and every second is another second in which a timeout somewhere can lose work you have already half done. Put it on a queue, return quickly, and let a worker handle it with its own retries.

6. A retry and something that costs money: add an idempotency key

Networks lose responses, not just requests. When a payment call times out, the client cannot tell a failed charge from a successful one it never heard about. The fix is to let the caller send a unique key and have the server return the original result for any repeat. Stripe's API documentation describes exactly this: it saves the status code and body of the first request for a given key, and subsequent requests with the same key return the same result. Once you have that, double charging stops being possible rather than merely unlikely.

7. A list endpoint and a growing table: paginate from day one

Every unbounded list is a future incident with a known date. Prefer cursor (keyset) pagination over offsets once data is large, because an offset query still has to walk past every skipped row, so it gets slower the deeper anyone goes, and it can skip or repeat rows when records are inserted mid-scan.

8. Expensive reads and repetition: a cache, plus a plan for invalidating it

The cache is the easy half. Before you add one, decide how an entry becomes wrong and what happens then. A cache without an invalidation story does not fail loudly. It quietly serves the wrong answer to a fraction of your users, indefinitely, and nobody notices until a customer does.

How backends fail, and how to fail well

9. Any network call with no timeout: set one

Many default HTTP clients will wait far longer than you want. The Python requests documentation is explicit that requests do not time out unless a timeout value is set explicitly. One slow dependency then holds every worker you have, and a service that was merely degraded takes down one that was healthy. A timeout on every outbound call is the highest-return resilience change available, and it is usually one line.

10. A failure and a retry: back off, add jitter, cap the attempts

Immediate, uniform retries are how a brief blip becomes a sustained outage, because every client comes back at the same instant. AWS's analysis of retry strategies concluded that jittered backoff should be considered a standard approach for remote clients. Retry only what is safe to repeat, which loops straight back to item six.

11. Traffic spikes and a shared resource: rate limits and backpressure

Deciding what to shed under load is a design decision, and you should make it calmly and in advance. If you do not, the system makes it for you at random, and random is the worst possible policy for choosing which users to disappoint. Rate limit per client, queue with a bounded size, and return a clear 429 rather than timing out.

12. An incident and a question about what happened: request ids in structured logs

Generate one identifier at the edge, pass it through every service, and put it in every log line as a structured field. Then a single user complaint can be traced end to end. Pair that with the four golden signals from Google's SRE book, latency, traffic, errors and saturation, and you can tell what is wrong before you know why. Add both before you need them, because the middle of an incident is when you will most regret their absence.

Backend mistakes that are expensive to undo

13. A secret and a repository: environment variables, and rotate anything committed

Assume anything that reached a commit is public forever, because history survives after the file is deleted. GitHub's own guidance says that for a leaked credential, the first step is to revoke or rotate the secret, before any history rewriting. Tools like Gitleaks catch secrets before they land.

14. A password and storage: a slow, salted hash

Never a general-purpose hash like SHA-256, never anything reversible. The OWASP Password Storage Cheat Sheet recommends Argon2id first, scrypt if Argon2id is not available, and bcrypt only for legacy systems. Inventing your own approach here is not merely unwise. The correct implementation is one library call, so there is no excuse for anything else. The wider OWASP Cheat Sheet Series is worth bookmarking for the rest of your security defaults.

15. A backup nobody has restored: perform the restore

A backup you have never restored is a belief rather than a backup, and the moment you discover it does not work is always the worst available moment. Restore it into a scratch database once, write down how long it actually took, and put the next drill in the calendar. There is even a backup and restore drill FYP idea built around exactly this.

16. A scaling problem and an instinct: find the bottleneck first

Adding servers to a system limited by a single database connection pool achieves nothing except a bigger bill. Measure, find the actual constraint, fix that one thing, then measure again, because the bottleneck moves every time you relieve it. A load test with a realistic mix of traffic tells you more than any amount of guessing; the URL shortener with a load test report is a good small project for practising this.

What else belongs on a backend development best practices list

A few that nearly made the sixteen:

• Events instead of long synchronous chains. Once you have more than a couple of services, a chain of five calls has roughly the availability of its worst link and the latency of all of them combined. I cover this in microservices practices that keep them workable.
• Health checks that check something real. An endpoint that returns 200 unconditionally tells your load balancer nothing.
• Connection pool limits set deliberately. They turn out to be the binding constraint far more often than anyone expects.
• Tests at the right layer. Integration tests against a real database catch problems mocks hide. See five kinds of API test.

The one I would put above all of those is knowing what your system is allowed to lose. Some data can be regenerated, some can be lost for a minute, and some can never be lost at all. Teams that never have that conversation end up protecting all three equally, which means overpaying for the cheap data and underprotecting the expensive kind.

How to learn these as a student or junior developer

You will not absorb sixteen defaults by reading a list. They stick when you have been bitten, and the good news is you can arrange to be bitten cheaply.

• Build one real backend and break it on purpose. Deploy a small API with a database, then kill the database mid-request, send the same POST twice, and load a table with a million rows. Watch what happens.
• Read the query plan for every slow query. It is the fastest way to understand indexing.
• Learn the fundamentals behind the defaults. Grokking System Design Fundamentals explains caching, queues, replication and consistency, which is the theory most of this list rests on. Hussein Nasser's backend engineering material is excellent and free.
• Read a large codebase. Seeing how mature projects handle retries and migrations teaches more than tutorials. My answer on how to read and understand a large codebase has a method.
• Keep the free references close. The Twelve-Factor App for configuration and deployment, and Designing Data-Intensive Applications when you want the deep version.

If you are choosing between backend and other tracks in the first place, how to choose between frontend, backend, mobile, cloud, security, data and AI might help.

The short version

Real database, correct indexes, transactions where they matter, migrations that expand before they contract. Move slow work off the request. Make retries safe with idempotency keys. Paginate everything. Cache with an invalidation plan. Timeouts on every call, backoff with jitter on every retry, rate limits before you need them, a request id in every log line. Secrets out of the repository, passwords through a slow hash, a restore you have actually performed. Measure before you scale.

Nothing on that list is clever, and that is the point. Backend engineering rewards the boring correct move made consistently far more than anything ingenious, because the systems that stay up are the ones where ordinary failures were handled ordinarily.

Learn it on Sefism

• Backend roadmap and the backend cheat sheet
• System design roadmap for the next layer up
• DevOps and cloud cheat sheet for deployment and operations basics
• Grokking System Design Fundamentals for caching, queues and consistency
• Grokking Database Fundamentals for Tech Interviews for transactions and indexing
• Web development FYP ideas to practise on a real project

FAQ

What are the most important backend development best practices?

Timeouts on every network call, idempotent writes, transactions where partial failure is unacceptable, pagination on every list, and backups you have actually restored. Those five prevent a large share of real incidents.

What are common mistakes backend developers make?

Doing slow work inside the request, retrying without backoff, caching without an invalidation plan, committing secrets, and scaling servers before finding the real bottleneck.

What is an idempotency key and why does it matter?

A unique value the client sends with a request so the server can recognise a retry and return the original result instead of doing the work again. It is what makes retrying a payment safe.

Should I use SQL or NoSQL for my backend?

Start with a relational database unless you have a specific reason not to. It handles most workloads well, gives you transactions, and is easier to reason about as the data grows.

How do I become a better backend developer?

Build and deploy something real, then deliberately break it. Learn the fundamentals behind caching, queues and databases, and read how mature codebases handle failure.

Ask your own question

If your situation is different from the ones above, the career questions library has honest answers to the questions students and engineers ask me most.

Want a learning roadmap built around you, written answers to your own questions and a place to discuss your FYP? Create your free Sefism account. Members can also book a one to one session when they want to talk it through.

Stay in touch

• LinkedIn: where I share roadmaps, career advice and honest takes on hiring. Follow, or message me.
• X (Twitter): short practical thoughts on engineering and careers.
• Instagram: quick tips and a more personal side.
• Sefism Discord: meet other students and engineers working through the same job market and share what is working.
• Sefism on YouTube: walkthroughs and talks.