Evidence-Grounded Pull Request Review Agent
AI code reviewers are easy to build and mostly useless, because they produce confident comments about problems that do not exist and developers stop reading them. Build one that must ground every comment in specific evidence from the codebase, verify each finding before posting it, and measure its false positive rate against real merged pull requests. The project is about earning trust in an automated reviewer, not about generating comments.
Advanced
Needs a team, real planning, and a supervisor who knows the area.1 semester · 2 to 3 students
7 to ship
3 optional extrasSuggested stack
What you should ship
- A GitHub Action that reviews a pull request and posts inline comments
- A retrieval step that pulls in definitions and call sites from outside the diff, so the reviewer sees context the diff does not show
- A verification pass where a second call must confirm a finding against quoted code before it is posted
- A labelled evaluation set of at least 100 pull requests from open source repositories, with known outcomes
- Precision and recall measured against that set, reported per finding category
- An ablation study showing what verification and retrieval each contribute to precision
- A configurable confidence threshold, with the precision and volume trade off documented
If you have time left
- Grouping related findings across files into one comment
- Learning from resolved comments so repeated false positives are suppressed
- Support for a locally hosted open model, with the quality difference measured
The problem
An automated reviewer that is wrong a third of the time is worse than no reviewer. Developers learn to skim past it, and the real findings get skipped along with the noise.
So the target is not more comments. It is precision high enough that people keep reading.
What makes this a real project
- Evidence grounding. Every finding must quote the code that proves it. A claim the model cannot support with a specific line does not get posted. This one rule removes a lot of the noise.
- Verification as a separate pass. The call that found the issue is a bad judge of it. A fresh call, given only the claim and the surrounding code, is a much better filter.
- Context beyond the diff. Most real bugs are about the interaction between changed code and code that did not change. If your reviewer only sees the diff, it will only find style issues.
- An honest evaluation set. Take merged pull requests, and use follow up fix commits as ground truth for what was actually wrong. Label them before you look at what your tool says.
Suggested approach
Build the evaluation set first. It is the slowest part and it decides whether any of your numbers mean anything.
Then build the simplest possible reviewer and measure it. Add retrieval, measure again. Add verification, measure again. Those three numbers are your ablation study and the core of your write up.
How you will know it worked
Run it on a repository that a real team maintains, and ask them to mark each comment as useful or noise. If more than one comment in five is noise, keep working.
Ideas and guidance, not finished projects
These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.
Not sure this one fits you?
I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Every session is free; a few slots open each week.
Follow along
New writing, resources and project ideas land here first.