Sefism is free for learners. We are here to help you grow, one step at a time.Join for free
Sefism.
AI & ML

Evidence-Grounded Pull Request Review Agent

An AI pull request reviewer for small engineering teams and open-source maintainers that must quote code evidence for every comment, verify each finding in a second pass and measure its false positive rate on real merged pull requests. An advanced final year project about earning developer trust, not generating more comments.

Topics
Difficulty

Advanced

Needs a team, real planning, and a supervisor who knows the area.
Effort

2 semesters · 2 to 3 students

Deliverables

7 to ship

4 optional extras

Suggested stack

PythonGitHub Actions and the GitHub APITree-sitterPostgreSQLAn LLM APIAn open model via Ollama
A suggestion, not a requirement. Swap anything for what you already know.

What you should ship

  • A GitHub Action that reviews a pull request and posts inline comments, each quoting the lines that support it
  • A retrieval step that pulls in definitions and call sites from outside the diff using Tree-sitter, so the reviewer sees context the diff hides
  • A verification pass where a fresh model call must confirm each finding against quoted code before it is posted
  • A labelled evaluation set of at least 100 pull requests from open-source repositories, using follow-up fix commits as ground truth
  • Precision and recall on that set, reported per finding category such as null handling, error handling and concurrency
  • An ablation study showing what retrieval and verification each contribute to precision
  • A configurable confidence threshold with the precision versus comment-volume trade-off documented

If you have time left

  • Grouping related findings across files into one comment
  • Learning from dismissed comments so repeated false positives are suppressed per repository
  • Running fully on a locally hosted open model, with the quality difference measured
  • A pilot with one real team who rate every comment as useful or noise for four weeks

The problem with AI code reviewers

AI code reviewers are easy to build and often ignored. They post confident comments about problems that do not exist, developers learn to skim past them, and the real findings get skipped along with the noise. An automated reviewer that is wrong a third of the time can be worse than no reviewer at all.

Small teams and open-source maintainers feel this most. They have the least review capacity, so a trustworthy second pair of eyes is valuable, and a noisy one wastes the scarce time they have. The target is not more comments. It is precision high enough that people keep reading.

What already exists and the gap

  • CodeRabbit reviews pull requests with a model plus layered linters and static analysis, and is generally described as precision-first.
  • Greptile indexes the whole repository into a graph and reviews each pull request against it, trading more noise for higher bug catch rates.
  • GitHub Copilot code review adds model-written review comments directly in GitHub.
  • Qodo focuses on review and test generation.

The honest gap: almost every published accuracy benchmark in this space is written by a company selling a reviewer. An open, reproducible study of what actually drives precision (retrieval, verification, thresholds) on a public labelled set is missing, and a student team is well placed to produce it.

What you build

  • Reviewer action: runs on each pull request, reads the diff, retrieves context, drafts findings.
  • Evidence rule: every finding must quote the specific lines that prove it. A claim the model cannot support with code is dropped.
  • Verifier: a separate call receives only the claim and the quoted code and must agree before posting.
  • Evaluation harness: replays historical pull requests and scores the reviewer against known outcomes.
  • Dashboard or report: precision, recall and volume per configuration.

The technically interesting part

Two problems make this FYP-worthy.

Context beyond the diff. Many real bugs live in the interaction between changed code and code that did not change: a function now returns null in one path, and three callers do not check. Use Tree-sitter to find definitions and call sites of every changed symbol, and give the reviewer those, not the whole repository.

Ground truth. Take merged pull requests where a later commit fixed a bug in the changed lines. The fix tells you what was actually wrong. Label them before you look at what your tool says, and also label clean pull requests so you can measure false positives.

How to evaluate it

  • Precision and recall on at least 100 labelled pull requests, per category.
  • Ablation: baseline reviewer, plus retrieval, plus verification. Three rows that form the core of your write-up.
  • Threshold curve: comments posted per pull request against precision.
  • Human rating: run it on a repository a real team maintains and have them mark each comment useful or noise. If more than one in five is noise, keep working.
  • Cost and latency per review, so a team can judge if it is worth running on every push.

Suggested plan

  1. Weeks 1 to 5: build the evaluation set first. It is the slowest part and decides whether any number means anything.
  2. Weeks 6 to 9: simplest reviewer as a GitHub Action, measured.
  3. Weeks 10 to 14: Tree-sitter retrieval, measured again.
  4. Semester 2, weeks 1 to 5: verification pass and evidence rule, measured again.
  5. Weeks 6 to 10: threshold tuning, open model comparison, real-team pilot.
  6. Weeks 11 to 14: report, demo and open-source release.

Path to a product

The first users are open-source maintainers, who can install a free GitHub Action and give public feedback, and local software houses that want a cheap first-pass reviewer on client repositories. Keep the core open source and charge teams for a hosted version with private repository support. Even if it never sells, a published precision study with a working Action is a strong portfolio piece for developer tooling and AI platform roles.

Scope warning

  • Support one or two languages, not all of them.
  • Do not try to auto-fix code; reviewing is the project.
  • Do not build a style linter; existing linters do that well.
  • Do not report accuracy on examples you looked at while building the prompts. Keep a held-out set.

Learn what you need on Sefism

Get help with this idea

Most teams attempting this skip the evaluation set and end up with a demo instead of a result. Tauseef can help you plan the labelling, pick repositories and design the ablation so your numbers hold up in front of examiners. If you are new here, create a free account first.

Ideas and guidance, not finished projects

These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.

Not sure this one fits you?

I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.

Discuss this idea
Work with me

Stuck on something specific?

Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.

Book a free sessionSee all services

Follow along

New writing, resources and project ideas land here first.