Flaky Test Detector for GitHub Actions That Uses Change Coverage Instead of Reruns
Build a CI tool that decides whether a failing test is flaky without rerunning it, by checking whether the failing test executed any of the code the commit changed. A test that fails without touching the change is almost certainly flaky. The tool labels failures on pull requests, tracks flaky tests over time and saves the CI minutes that blanket reruns waste.
Intermediate
A small team, or one strong student willing to learn something new.1 to 2 semesters · 2 students
7 to ship
4 optional extrasSuggested stack
What you should ship
- A diff reader that lists changed lines and functions for each commit or pull request
- Lightweight coverage collection limited to changed files, to keep overhead low
- A verdict per failing test: likely real (touched the change), likely flaky (did not), or unknown
- Pull request comments and check annotations that explain each verdict
- A history store with flakiness rates per test and an optional quarantine list
- A measurement of overhead and accuracy on at least five open source projects
- A comparison with rerun-based detection on the same failures
If you have time left
- Root cause hints: test order dependence, time, network or concurrency
- Support for a second language and test runner
- Automatic issue creation for tests that cross a flakiness threshold
- Detecting tests made flaky by shared state with the previous test
The problem: red builds nobody believes
A flaky test passes and fails on the same code. Every growing project has them: tests that depend on time, ordering, network or timing between threads. When a build goes red, developers first ask "is it real or is it flaky?" and often just press rerun. Over time people stop trusting failures, and real bugs get merged behind a "probably flaky" shrug.
The usual fix is to rerun failing tests several times and call them flaky if they eventually pass. That costs CI minutes, slows every pull request, and still misses tests that fail rarely. Teams on free CI tiers or paying for minutes feel this directly.
What already exists and the gap
- Trunk Flaky Tests and BuildPulse detect flaky tests from pass and fail history and can quarantine them, as hosted services.
- Test runner plugins such as pytest-rerunfailures retry failures automatically.
- CI platforms show test history but leave the judgement to developers.
These rely on history and reruns. The gap is an open, self-hostable tool that gives a verdict on the first failure, without rerunning, and explains why: "this test did not execute any code you changed".
Inspiration
"DeFlaker: Automatically Detecting Flaky Tests" by Jonathan Bell, Owolabi Legunsen, Michael Hilton and colleagues (ICSE 2018) marks a newly failing test as flaky if it did not execute any of the latest code changes. In their experiments it found 1,874 flaky tests out of 4,846 failures with a 1.5% false alarm rate, with very low overhead, and far better recall than Maven's rerun-based detector. DeFlaker targeted Java. This project brings the idea to Python and JavaScript projects on GitHub Actions.
What you build
- Developers see on their pull request: "test_checkout_total failed; it ran 3 of your changed lines; this looks like a real failure", or "test_email_retry failed without touching your change; likely flaky (failed 6 times in 200 runs on main)".
- Maintainers get a ranked list of the flakiest tests and can quarantine them.
- Teams see CI minutes saved compared with automatic reruns.
The technically interesting part
- Cheap coverage. Full coverage slows tests down. Instrument only the files a change touched, and only record whether changed lines ran, which costs far less.
- What counts as the change. Lines, functions, configuration files and test fixtures can all change behaviour. Decide how to handle changes to non-code files, and when to answer "unknown" instead.
- Combining signals. Change coverage gives a verdict per failure; history gives a rate per test. Combine them so a test that touched the change but has a long flaky history is still treated with care.
- Fitting into CI. Collect coverage and results inside the job, send them to a small service, and post the verdict back as a check annotation within seconds.
- Order and shared state. Some tests fail only when another test runs first and leaves state behind. Record test order with each run so that, when a failure looks flaky, you can point at the likely polluting test rather than just saying "flaky".
- Quarantine without hiding. Quarantined tests should still run and report, so a real regression in a flaky area is not silently ignored. Show quarantined failures separately, with a reminder of how long the test has been quarantined.
How to evaluate it
- Ground truth: for a sample of failures in open source projects, rerun many times and inspect by hand to label them real or flaky. Compare your verdicts with these labels.
- Overhead: test suite time with and without your instrumentation.
- CI minutes: minutes that reruns would have used against what your approach uses.
- Developer view: a short survey of developers on a project that tries it.
Suggested plan
- Weeks 1 to 3: read DeFlaker; collect flaky failures from two open source projects.
- Weeks 4 to 7: diff reader and changed-line coverage for pytest.
- Weeks 8 to 11: verdict logic, GitHub check annotations and history store.
- Weeks 12 to 15: evaluation, dashboard and report.
- Optional second semester: Jest support and root cause hints.
Path to a product
Flaky tests are a well known cost and paid products exist, which shows demand. An open source GitHub Action with a hosted dashboard for private repositories is a familiar model. The evaluation also makes a good portfolio piece for platform and developer productivity roles.
Scope warning
Support one language and one test runner well before adding another. Do not try to fix flaky tests automatically; explaining them is enough for a strong project.
Learn what you need on Sefism
- QA and test automation roadmap: test design and why tests become flaky.
- QA and test automation cheat sheet: test types and CI practice in one place.
- GitHub Actions Documentation: checks, annotations and artifacts.
- pytest Documentation: plugins and hooks for collecting results.
- Platform engineering and SRE roadmap: developer productivity as a platform concern.
Get help with this idea
The hard part is cheap coverage and an honest evaluation with good ground truth. Tauseef can help you pick projects to study, design the labelling process and keep the scope to one language. If you are new here, create a free account first.
Ideas and guidance, not finished projects
These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.
Not sure this one fits you?
I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
Follow along
New writing, resources and project ideas land here first.