Chapter 2 · Practice 2 of 4
Use counters for things that only go up, gauges for everything else
Why it matters
The Prometheus instrumentation guide's rule is simple: if the value can go down, it is a gauge. Counters only increase, and reset to zero when the process restarts.
| Type | Examples | How you query it |
|---|---|---|
| Counter | Requests served, errors, bytes sent | rate() or increase() |
| Gauge | Queue length, in-flight requests, memory in use | The value itself, avg_over_time() |
| Histogram | Request duration, payload size | histogram_quantile() and bucket counts |
Two more tips from the guide:
- Never apply
rate()to a gauge. - Export the timestamp of an event (such as the last successful run) rather than the time since it happened, and compute the age in the query with
time() - metric.
For batch jobs, the guide says the key metric is the last time it succeeded. Alert when that is too old, not on individual failures.
Example
# Alert when the nightly export has not succeeded for 26 hours
time() - export_last_success_timestamp_seconds > 26 * 3600