Sefism is free for learners. We are here to help you grow, one step at a time.Join for free
Sefism.
Cloud & DevOps

Mutation Testing Bot That Grades AI-Written Tests on Pull Requests

Build a pull request bot that runs mutation testing only on the code a pull request changed, then comments with the mutants its tests failed to catch. AI tools write tests quickly, and many of them check nothing. This bot shows reviewers which new tests are decoration, in minutes rather than the hours a full mutation run takes.

Topics
Difficulty

Intermediate

A small team, or one strong student willing to learn something new.
Effort

1 semester · 2 students

Deliverables

7 to ship

4 optional extras

Suggested stack

Python or TypeScriptStryker, mutmut or PITGitHub Actions and the GitHub REST APIDockerA small web dashboard (optional)
A suggestion, not a requirement. Swap anything for what you already know.

What you should ship

  • Diff analysis that maps a pull request to the changed functions and the tests that cover them
  • Mutation runs limited to changed code, using an existing engine for at least one language
  • A pull request comment listing surviving mutants with the exact change and the line, grouped by test file
  • A mutation score for the changed code with a configurable threshold that can fail the check
  • Caching so unchanged code is not mutated again on the next push
  • A benchmark on real open source pull requests comparing diff-based runs with full runs
  • A study of AI-generated versus hand-written tests on the same functions, with surviving mutant counts

If you have time left

  • Support for a second language and mutation engine
  • Suggested test cases for each surviving mutant, marked clearly as suggestions to review
  • Flaky test detection so a random failure is not counted as a killed mutant
  • A dashboard of mutation scores per repository over time

The problem: tests that pass whatever the code does

AI coding tools can write fifty tests in a minute. Many of them look thorough and check nothing: they assert that a mock was called with the values the test passed in, snapshot output nobody reviews, or only check that a function returned something. Coverage numbers go up, so the pull request looks well tested. Reviewers rarely have time to read every test, and the bugs slip through.

Mutation testing answers the real question: if the code were wrong, would a test fail? It makes small deliberate bugs, such as flipping > to >=, and reruns the tests. A mutant that survives shows a behaviour no test checks. The catch is speed: mutating a whole codebase can take hours.

What already exists and the gap

  • Stryker (JavaScript, TypeScript, C# and Scala), mutmut (Python) and PIT (Java) are mature open source mutation testing tools. Stryker has an incremental mode that reuses earlier results.
  • Coverage services report which lines ran, not whether results were checked.
  • AI code review tools comment on code quality but do not measure test strength.

The gap is a bot that runs mutation testing only on what a pull request changed, fast enough for every push, and reports surviving mutants where reviewers already look: in the pull request.

Inspiration

Google's paper "State of Mutation Testing at Google" (ICSE SEIP 2018) describes surfacing a small number of mutants during code review, only on changed lines, so developers see them while the change is fresh. This project brings that idea to small teams and open source projects, with a focus on tests written by AI tools.

What you build

  • Developers push a pull request and get a comment: the mutation score of the changed code and the mutants that survived, each with the exact change and line.
  • Reviewers use the comment to ask for real tests instead of reading every test file.
  • Maintainers set a threshold that fails the check when the changed code is poorly tested.

The technically interesting part

  • Mapping a diff to tests. Find changed functions from the diff, then the tests that cover them, using coverage data from a single instrumented run.
  • Keeping it fast. Mutate only changed lines, run only relevant tests, run mutants in parallel and cache results between pushes.
  • Handling noise. Equivalent mutants (changes that do not alter behaviour) and flaky tests both distort scores; detect and report them rather than hide them.
  • The study. Take functions from real projects, generate tests with an AI tool and write tests by hand, then compare surviving mutants. Report what kinds of weak test AI produces most often.

How to evaluate it

  • Speed: time per pull request against a full mutation run on the same repository.
  • Agreement: mutants found by the diff-based run that a full run also finds.
  • Usefulness: on real open source pull requests, how many surviving mutants pointed to a real missing test, judged by hand.
  • The AI test study: surviving mutants per function for AI-written and hand-written tests.

Suggested plan

  • Weeks 1 to 3: learn one mutation engine, run it on a sample project, measure a full run.
  • Weeks 4 to 7: diff mapping, selective runs and caching.
  • Weeks 8 to 10: GitHub Action, pull request comments and thresholds.
  • Weeks 11 to 13: benchmark on open source pull requests and the AI test study.
  • Weeks 14 to 15: report and demo.

Path to a product

Test quality is a growing worry as more code is AI-written, and coverage is a poor proxy. A free GitHub Action for open source with a paid dashboard for private repositories is a familiar model for developer tools. The study itself is also publishable and makes a strong portfolio piece for quality and developer tooling roles.

Scope warning

Do not write your own mutation engine; wrap an existing one. Support one language well before adding another. Generated test suggestions are a stretch goal, and they must be clearly labelled as suggestions.

Learn what you need on Sefism

Get help with this idea

The risk is a slow bot nobody waits for, or a noisy one nobody trusts. Tauseef can help you scope the diff mapping, design the benchmark and structure the AI test study. If you are new here, create a free account first.

Ideas and guidance, not finished projects

These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.

Not sure this one fits you?

I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.

Discuss this idea
Work with me

Stuck on something specific?

Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.

Book a free sessionSee all services

Follow along

New writing, resources and project ideas land here first.