Alert Noise Analyser and Replay Tool for Small On-Call Teams

The problem: pages that teach people to ignore pages

A small team adds alerts one incident at a time: CPU high, disk filling, error rate up, latency up. A year later the on-call phone buzzes several times a night, mostly for things that fix themselves in minutes or that nobody acts on. People start silencing notifications, and the one alert that matters is missed. Large companies have SRE teams to tune alerting. Small teams have a group chat and tired engineers.

Nobody tunes alerts because nobody knows which change is safe. Raise a threshold and you might miss the next outage.

The cost is real even for a team of five. Broken sleep makes people slower and more likely to make mistakes the next day, and engineers leave jobs where on-call is miserable. Meanwhile the alert history that could guide better decisions already exists in Alertmanager; it simply is not analysed. Turning that history into a few clear numbers and a safe way to test changes is a small amount of engineering for a large improvement in team health.

What already exists and the gap

• Prometheus Alertmanager groups, inhibits and silences alerts with static rules that a person must write.
• Grafana alerting offers similar routing and grouping.
• Commercial incident platforms such as PagerDuty offer noise reduction features, typically on paid plans and for teams already using them.

The gap is an open, simple analyser for teams on the Prometheus stack that measures noise from their own history and, most importantly, replays that history against a proposed change, so the team can see the trade-off before making it.

Inspiration

Google's Site Reliability Engineering book argues that every page should be actionable and urgent, and that pages which are routinely ignored should be removed or changed. On the research side, Identifying Linked Incidents in Large-Scale Online Service Systems (Chen et al., ESEC/FSE 2020, from Microsoft and university collaborators) studies linking related incidents so engineers see one problem rather than many. This project applies simple, explainable versions of these ideas to a small team's alert history.

What you build

• On-call engineers get a weekly report: the noisiest alerts, how many night pages each caused and how often they were resolved without action.
• The team opens a dashboard, picks a noisy alert and tries a change, such as "fire only if true for 10 minutes". The replay shows: 34 fewer pages last month, and the two real incidents would have paged 4 and 7 minutes later.
• Grouped incidents show alerts that fire together, suggesting inhibition rules so one problem pages once.

The technically interesting part

• Defining noise. Turn vague complaints into metrics: flapping (fires and resolves repeatedly), self-resolving, never acknowledged, fired outside working hours. Decide what counts as acted on using acknowledgements, linked incidents or silence events.
• Grouping. Alerts from one root cause share time and often labels such as service or host. Start with time windows and label overlap; measure how well groups match the incidents you injected in the lab.
• Replay. Re-evaluating a threshold change needs the underlying metric, not just the alert. Store or query the metric history from Prometheus and re-run the rule logic, including for durations, to produce a new alert history.
• Ground truth. In the lab, you know exactly when real faults were injected. Use that to measure whether a proposal delays or misses real incidents.

How to evaluate it

• Lab experiments: run the demo application for several weeks of simulated time with injected faults and natural noise. For each proposed change, measure pages saved and detection delay for real faults.
• Grouping quality: how often grouped alerts correspond to a single injected fault.
• Real history: with permission, analyse one real team's alert history and review findings with them.
• Usability: can an engineer who did not build the tool propose and evaluate a change in under 15 minutes?

Suggested plan

• Weeks 1 to 3: set up the lab with Prometheus, Alertmanager, Grafana and a demo service; define noise metrics.
• Weeks 4 to 7: collector, storage and noise metrics.
• Weeks 8 to 10: grouping and the replay engine.
• Weeks 11 to 13: dashboard, weekly report and experiments.
• Weeks 14 to 15: real team analysis, report and demo.

Path to a product

On-call burnout is a well-known problem, and many small teams use the Prometheus stack without a paid incident platform. An open source analyser with a hosted option is a realistic route, and the replay idea is a clear differentiator. The project is strong evidence for SRE and platform engineering roles.

Scope warning

Do not build an alert router or a paging system; analyse and replay. Start with Prometheus and Alertmanager only. Recommendations must always show the trade-off and never apply changes automatically.

Learn what you need on Sefism

• Platform engineering and SRE roadmap: alerting, on-call and reliability practice.
• Google Site Reliability Engineering book: the chapter on monitoring distributed systems.
• Prometheus Documentation: alerting rules and the query API.
• Implementing Service Level Objectives: replacing noisy alerts with SLO-based ones.
• DevOps and cloud cheat sheet: Docker Compose and monitoring basics for the lab.

Get help with this idea

The hard parts are a realistic lab and a replay that is trustworthy. Tauseef can help you design the fault injections, define the noise metrics and present the results for SRE roles. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session