Observability Best Practices

Chapter 2 · Practice 3 of 4

Keep label cardinality low, and never label by user or request

From Metrics that stay useful and affordable

Why it matters

Every unique combination of label values is a separate time series, with its own storage and memory. One unbounded label can multiply your bill or take down the metrics backend.

The Prometheus guides give concrete limits:

  • Do not use labels for high-cardinality dimensions such as user IDs, email addresses or other unbounded sets of values.
  • Keep the cardinality of each metric below about 10 as a rule, with only a handful of metrics above that across the whole system.
  • Anything over 100, or that could grow that large, needs a different approach.
  • Most metrics should have no labels; add them when there is a concrete need.

High-cardinality detail belongs in logs and traces, which are built to store per-request data. Metrics answer "how many and how fast"; traces and logs answer "which one".

To check, query your backend for the series count per metric name, and investigate the top few.

Avoid

  • Labels such as user_id, email, request_id or full URLs
  • Raw paths like /users/123 as a label value; use the route template

Example

PROMQL
# Top 10 metric names by series count
topk(10, count by (__name__) ({__name__=~".+"}))