Microservices Practices That Actually Keep Them Workable

The short answer on microservices best practices

Most lists of microservices best practices explain how to run services well and skip the decision that causes the most damage: splitting too early. So my first practice, and the one that saves the most pain, is simple. Do not start with microservices unless you have the problem they solve.

I am a Lead Full Stack Engineer and I have worked on both sides of this: large single applications that were a pleasure to change, and distributed systems where adding one field meant three coordinated deployments. The difference was never the technology. It was whether the boundaries matched the business and whether the team respected them.

This post covers when microservices make sense, and then the practices that actually keep them workable once you have them: boundaries, data ownership, failure handling, idempotency, events, observability, contract testing and deployment. If you are a student, most of this will feel abstract until your first job. Read it anyway, because these ideas appear in every system design conversation you will ever have.

Before the best practices: should you use microservices at all?

A well-structured single application with clear internal modules is the right answer for most teams for a long time. Microservices solve an organisational problem, many teams needing to ship independently, and they solve it by adding a distributed systems problem. If you do not have the first problem, you are buying the second for nothing.

Martin Fowler made the same observation years ago in MonolithFirst: almost all the successful microservice stories he had heard of started with a monolith that got too big and was broken up, and systems built as microservices from scratch often ended up in serious trouble.

There is a well-known recent example in the other direction too. In 2023 the Prime Video team wrote that they moved a video quality monitoring service from a distributed serverless design into a single application and cut infrastructure costs by around 90%. That was one service, not all of Prime Video, but the lesson holds: distribution has a price, and you should pay it only when it buys you something.

If most of your answers sit in the left column, build a modular monolith: one deployable, with internal boundaries you would be happy to split along later. If you already have services, or genuinely have the organisational problem, the rest of this is what keeps them healthy.

Microservices best practices for boundaries and data

Draw boundaries around business capabilities, not technical layers

The most common bad split is by layer: one service for data access, one for the API, one for the logic. Every real change touches all three, so you now need three coordinated deployments to add a field.

Split by business capability instead: orders, payments, inventory, notifications. The test is whether a typical feature fits inside one service. If ordinary work keeps spanning four services, the boundaries are wrong and no amount of tooling will fix it.

It also helps to remember Conway's law: organisations tend to produce systems that mirror their communication structures. If two teams share a service, expect that service to become a negotiation. Align boundaries with team ownership where you can.

Every service owns its data, and nothing else touches it

One database, or at least one schema, per service, and no other service reads those tables directly. The database per service pattern keeps each service's data private and reachable only through its API.

This is the practice teams break first, because reaching into another service's tables is so much easier than asking properly. The moment it happens, you no longer have two services. You have one system with two pipelines and a shared schema nobody can change safely. If a service needs another's data, it asks through the API, or subscribes to events and keeps its own copy.

The cost is real: cross-service queries and transactions become harder, and you will meet patterns like sagas for multi-step business processes. That cost is exactly why you should only split when you must.

Resilience: assume every call fails

A function call becomes a network call, and network calls fail slowly, which is worse than failing quickly. Four defaults cover most of it:

• Timeouts on everything. A call with no timeout eventually hangs, and hanging threads take the whole service down. Set them explicitly, per dependency.
• Retries with backoff and jitter, for the right errors only. Retry a timeout or a 503. Do not retry a 400, because it will be invalid the second time too. AWS's analysis concluded jittered backoff should be a standard approach for remote clients, because uniform retries turn a blip into a stampede.
• Circuit breakers. After enough failures, stop calling and fail fast. In Fowler's description of the circuit breaker, once the breaker trips, further calls return an error without the protected call being made at all, which stops one slow dependency dragging down everything upstream.
• Bulkheads. Separate connection and thread pools per dependency, so one saturated downstream cannot consume every resource you have.

I list the same defaults from a single-service angle in sixteen things that break a backend. In a distributed system they stop being good hygiene and become survival.

Make operations safe to repeat

This one deserves its own section because it is where real money is lost.

If a call times out, you do not know whether it ran. Retrying might duplicate it. For a payment or an order, that is an incident.

Give write operations an idempotency key: the client generates a unique identifier, the server records it, and a repeat with the same key returns the original result rather than doing the work twice. Stripe's API is the reference example: it stores the status code and body of the first request for a key and returns the same result for later requests with that key. Every write endpoint that matters needs this.

Prefer events for anything that can wait

Synchronous calls chain failures. If A calls B calls C and C is slow, A is slow, and a user is watching.

Where work does not need to happen before the response, publish an event and let interested services react. The order service records the order and emits OrderPlaced; email, analytics and inventory pick it up on their own time.

The trap is the dual write: updating your database and publishing an event as two separate steps, where the second can fail after the first succeeded. The standard fix is the transactional outbox: write the event into an outbox table in the same transaction as the business data, and have a separate relay publish from that table. Messages are then sent if and only if the transaction commits.

Assume messages arrive twice

The outbox relay can publish duplicates, and most brokers give you at-least-once delivery anyway. So consumers must handle repeats: record what you have processed, or make the operation naturally repeatable. This is idempotency again in a different costume, and it is the most transferable idea in this whole post.

Observability: you cannot operate what you cannot see

With one application you read the log. With twenty services you need to follow one request across all of them.

• Trace identifiers. Generate one at the edge, pass it through every call and into every log line. Without it, debugging across services is guesswork. Building distributed tracing for a polyglot microservice system is a strong final year project for exactly this reason.
• Structured logs. JSON with consistent field names across services. Grepping twenty different log formats does not work.
• The four golden signals per service. The Google SRE book names them: latency, traffic, errors and saturation. Track them consistently, in one place. The full Site Reliability Engineering book is free and worth reading.
• Health checks that check dependencies, rather than returning 200 unconditionally, so your orchestrator can act on them.

Test the contracts, not just the services

Unit tests in every service pass. The system is broken anyway, because one team renamed a response field.

Contract testing pins down what each service promises and fails the build when a change breaks a consumer. In consumer-driven contract testing, as the Practical Test Pyramid puts it, the consumers drive the implementation of the contract. Pact is the common open source tool, and it is far cheaper than end-to-end tests spanning every service. I go deeper on test types in five kinds of API test. A small API contract checker for CI is a good way to learn it hands on.

Deploy small and gradually

Each service deploying on its own schedule is the entire point. Use rolling or canary releases, watch the golden signals, and roll back automatically on regression. The mechanics are covered in what happens between your pull request and production.

If deploying one service requires coordinating with three other teams, you have a distributed monolith. It has all the operational cost of microservices and none of the independence, and it is the most common failure mode I see.

Microservices best practices checklist

How to learn microservices properly

Nearly everything above is a response to one fact: the network is unreliable. Services buy you independent deployment and team autonomy, and they cost you consistency, debuggability and a lot of operational work. That trade is worth it when coordination between teams is your bottleneck, and a bad trade otherwise.

To learn it well:

• Get the fundamentals first. Caching, queues, replication and consistency come before microservices. The system design roadmap puts them in order, and Grokking System Design Fundamentals covers the groundwork these practices assume.
• Study the patterns in a structured way. Grokking Microservices Design Patterns walks through sagas, outbox, circuit breakers and the rest with worked examples.
• Read the honest book. Building Microservices by Sam Newman is balanced about when not to use them. Martin Fowler's website has most of the canonical articles free.
• Build something small and distributed. Two services, a queue and a trace across them teaches more than any diagram. The cloud and DevOps FYP ideas have several options.

Learn it on Sefism

• System design roadmap and the system design cheat sheet
• DevOps and cloud roadmap for deployment and operations
• Grokking Microservices Design Patterns
• Grokking System Design Fundamentals
• Building Microservices and the wider system design resources
• A learning roadmap for system design

FAQ

What are the best practices for microservices?

Split by business capability, give each service its own data, put timeouts, retries and circuit breakers on every call, make writes idempotent, use events with an outbox for work that can wait, trace every request, and test contracts between services.

When should you not use microservices?

When you have one or two teams, a domain you are still discovering, or no CI/CD and observability in place. A modular monolith is simpler, cheaper and easier to change in those situations.

Should each microservice have its own database?

Yes. Each service should own its data and expose it only through its API or events. Sharing tables between services recreates the coupling you split to escape.

What is a distributed monolith?

A system split into services that still have to be deployed together or changed in lockstep. It carries the cost of microservices without the independence.

How do microservices communicate?

Synchronously over HTTP or gRPC when the caller needs an answer now, and asynchronously through message queues or event streams when the work can happen later.

Ask your own question

If your situation is different from the ones above, the career questions library has honest answers to the questions students and engineers ask me most.

Want a learning roadmap built around you, written answers to your own questions and a place to discuss your FYP? Create your free Sefism account. Members can also book a one to one session when they want to talk it through.

Stay in touch

• LinkedIn: where I share roadmaps, career advice and honest takes on hiring. Follow, or message me.
• X (Twitter): short practical thoughts on engineering and careers.
• Instagram: quick tips and a more personal side.
• Sefism Discord: meet other students and engineers working through the same job market and share what is working.
• Sefism on YouTube: walkthroughs and talks.