Chapter 2 · Practice 3 of 4
Fix or quarantine flaky tests immediately
Why it matters
A test that fails randomly teaches everyone to click "re-run" without reading the failure. After a few weeks, real failures get re-run too, and the suite stops protecting you.
Treat a flaky test as a production bug:
- When a test fails and then passes on retry with no code change, flag it.
- Move it out of the blocking suite into a quarantine job that still runs and reports.
- Assign an owner and a deadline; delete the test if nobody will fix it.
- Fix the usual causes: shared state between tests, real clocks and timeouts, test order, network calls.
Automatic retries can hide flakiness entirely. If your tool retries failed tests, make it report the retry so you still see the pattern.
To check, count re-runs per week in your CI system. A rising number is the clearest sign of flakiness.
Do
- Track tests that pass only on retry
- Give every quarantined test an owner and a deadline
Avoid
- Silent automatic retries with no report