Best practices · Cloud and DevOps
Observability Best Practices
Opens with creditsA practical guide to observability best practices for teams running services in production. It covers choosing what to measure with SLIs and SLOs, designing metrics that stay cheap, writing logs you can search and join to traces, instrumenting with OpenTelemetry, running a telemetry pipeline, and alerting on user pain instead of noise.
Written by Tauseef Fayyaz6 chapters, 23 practicesAbout 17 min22 quiz questionsReviewed 13 October 2026What you will learn
6 chapters · about 17 min
1Deciding what to measureStart from what users experience. These practices turn that into a small set of indicators and objectives that every other signal supports.4 practices · 3 min
2Metrics that stay useful and affordableHow to name, type and label metrics so queries are obvious and the time series count does not explode.4 practices · 3 min
- Logs you can search and joinHow to write logs that machines can filter, that connect to the trace of the same request, and that do not leak secrets or cost a fortune.4 practices · 3 minOpens with credits
- Distributed tracing with OpenTelemetryHow to instrument services so a single request can be followed end to end, using OpenTelemetry, its semantic conventions and sensible sampling.4 practices · 3 minOpens with credits
- The telemetry pipelineHow telemetry gets from your services to a backend: the OpenTelemetry Collector, resource attributes and protecting the pipeline itself.3 practices · 2 minOpens with credits
- Alerting that people trustHow to page only for real user impact, using SLO burn rates, and make every page something a human can act on.4 practices · 3 minOpens with credits
About this guide
Observability best practices exist to answer two questions quickly: is something wrong for users right now, and where is it coming from? Most teams collect plenty of data and still struggle with both. Dashboards show hundreds of graphs but no clear sign of user impact. Alerts fire on CPU spikes nobody needs to act on, so the real page at 3 a.m. gets ignored. Logs are free text that cannot be filtered, and the one request that failed cannot be followed from the load balancer to the database.
The fix is not more data, it is better chosen data. This guide starts with deciding what to measure, using service level indicators and objectives and the four golden signals from Google's SRE books. It then covers the three main signals in turn: metrics that follow the Prometheus naming and cardinality rules, structured logs that carry trace context, and distributed traces built with OpenTelemetry and its semantic conventions. The last chapters cover the collection pipeline (the OpenTelemetry Collector and sampling) and alerting on SLO burn rate, so pages are rare and always mean something.
Every practice was checked against primary sources: the Google SRE book and workbook chapters on monitoring, implementing SLOs and alerting on SLOs, the Prometheus instrumentation and naming guides, and the OpenTelemetry documentation. Numbers in this guide, such as burn rates and cardinality limits, come from those pages.
The practices are vendor neutral. They work the same whether your backend is Prometheus and Grafana, an open source tracing store, or a commercial observability platform, because OpenTelemetry lets you change the backend without re-instrumenting the code.
Common questions
What is the difference between monitoring and observability?
Monitoring usually means watching known signals and alerting when they cross thresholds. Observability is the wider ability to ask new questions about production from the telemetry you collect, such as following one slow request across services. In practice you need both: SLO-based alerts tell you something is wrong, and traces, logs and metrics tell you why.
What are the four golden signals?
Latency, traffic, errors and saturation, from the monitoring chapter of Google’s SRE book. If you can only measure four things about a user-facing system, measure these, and track latency for successful and failed requests separately.
What is an error budget burn rate alert?
It fires when you are spending your error budget faster than the SLO allows. The SRE workbook’s starting point for a 99.9% SLO pages at 14.4x burn over 1 hour (confirmed over 5 minutes) and 6x over 6 hours, and opens a ticket at 1x over 3 days.
Should I use OpenTelemetry?
For new instrumentation, yes in most cases. It is vendor neutral, auto-instrumentation covers most popular frameworks, and the Collector lets you change backends without re-instrumenting. Check the OpenTelemetry status page for your language, since signal maturity still varies.
How many labels should a Prometheus metric have?
As few as possible. The Prometheus instrumentation guide suggests keeping each metric’s cardinality below about 10, treating anything over 100 as a sign to find another approach, and never using unbounded values such as user IDs or emails as labels.
Do I need to sample traces?
Not at low volume. The OpenTelemetry docs suggest considering it around 1,000 or more traces per second. When you do, tail sampling lets you keep every error and slow trace while dropping routine ones.
Sources (8)
Checked against these, last on 13 October 2026.
- Monitoring Distributed Systems, Google SRE Booksre.google/sre-book/monitoring-distributed-systems/
- Implementing SLOs, Google SRE Workbooksre.google/workbook/implementing-slos/
- Alerting on SLOs, Google SRE Workbooksre.google/workbook/alerting-on-slos/
- Metric and label naming, Prometheusprometheus.io/docs/practices/naming/
- Instrumentation, Prometheusprometheus.io/docs/practices/instrumentation/
- Sampling, OpenTelemetryopentelemetry.io/docs/concepts/sampling/
- Collector, OpenTelemetryopentelemetry.io/docs/collector/
- OpenTelemetry semantic conventions, OpenTelemetryopentelemetry.io/docs/specs/semconv/
Related on Sefism
Picked from the other libraries by shared topics.
Roadmaps
See allCheat sheets
See allBest practices
See all- Kubernetes Best PracticesRun workloads on Kubernetes that schedule well, survive restarts and node drains, and stay locked down: resources, probes, PDBs, Pod Security and RBAC.
- Terraform Best PracticesWrite Terraform that teams can review and change safely: file layout, version pinning, remote state with locking, secrets, modules, refactoring and tests.
- CI/CD Best PracticesBuild pipelines that are fast, trustworthy and safe to deploy from: trunk-based flow, fast tests, locked-down workflows, signed artifacts, canaries and DORA metrics.
Checklists
See all- Production Readiness ChecklistCheck a service before it takes real traffic: ownership, SLOs, capacity, timeouts and retries, alerts, rollout and rollback, backups, security and on-call.
- Incident Response ChecklistPrepare for and handle production incidents: on-call, severity levels, declaring, roles, mitigation, security containment, status updates and blameless postmortems.
- Kubernetes Production ChecklistVerify a workload and its cluster before real traffic: requests and limits, probes, PDBs, spread, Pod Security, RBAC, network policies, secrets, images, autoscaling, upgrades and backups.
Project structures
See all- Spring Boot project structureA REST API in Java with Spring Boot 4, PostgreSQL, Flyway and Testcontainers, organised by feature the way the Spring docs suggest. It grows from the Initializr scaffold to a service with JWT security, problem-detail errors, MapStruct DTOs, slice and integration tests, CI, containers and observability, and finally Modulith boundaries, contract and load tests. For students who know some Java and want to see how a real service is organised.
- ASP.NET Core Web API project structureA REST API in C# on .NET 10 with minimal API endpoints grouped by feature, EF Core and PostgreSQL, JWT auth with policies, xUnit and Testcontainers tests, and everything production needs: health checks, OpenTelemetry, rate limits, versioning, caching and a CI pipeline. It shows the popular clean architecture split (Api, Application, Domain, Infrastructure) honestly, and at Expert grows into Aspire, a modular monolith and enforced boundaries.
- Go REST API project structureA JSON API in Go using the standard library router, PostgreSQL through pgx and sqlc, SQL migrations with goose, and tests next to the code. It follows the layout the Go team itself documents: commands in cmd/, everything else in internal/, one package per feature plus a small platform package for the plumbing, and nothing added until it is needed.
Career questions
See all- Do I need Docker and Kubernetes as a junior developer?Docker, yes: learn the basics, because nearly every team uses containers and you will be expected to run the project locally and read a Dockerfile. Kubernetes, mostly no: understand what it does and a few core terms, but deep Kubernetes can wait until your job or a DevOps direction needs it.
- What is platform engineering, and how is it different from DevOps?DevOps is a culture and set of practices: developers and operations sharing responsibility for shipping and running software. Platform engineering is one way to put that into practice at scale: a team builds an internal platform, a paved road of tools and templates, so product teams can ship without becoming infrastructure experts.
- Can I become an SRE or platform engineer straight after graduating?Sometimes, but it is rare, because these roles expect you to have run production systems. The more reliable path is a first job in backend, DevOps or cloud support, where you learn production, then a move into SRE or platform work within a couple of years.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
- 1:1 Mentorship
- Fresh Graduate Mentorship
- Career Guidance
Brand deals, promos and partnerships: tauseeffayyaz@gmail.com