Prompt Change Regression Evals in Pull Requests with Statistical Diffs
Teams change a prompt, swap a model or tweak retrieval and only find out a week later that something broke. Build a GitHub Action that detects AI-related changes in a pull request, runs the affected eval suites against the old and new versions, accounts for the randomness in LLM outputs with repeated runs and confidence intervals, and posts a clear diff of what got better, what got worse, and what it costs.
Intermediate
A small team, or one strong student willing to learn something new.1 semester · 2 students
7 to ship
3 optional extrasSuggested stack
What you should ship
- Change detection: map prompt files, model config and retrieval settings in a repo to the eval suites they affect
- A runner that evaluates base and head versions of each affected suite, with caching so unchanged cases are not re-run
- Repeated runs per case and a statistical comparison (paired test or bootstrap intervals) so random wobble is not reported as a regression
- A pull request comment showing pass-rate change with intervals, the specific cases that flipped, latency and token cost change
- A merge gate policy file: which suites must not drop and by how much
- A demo repository with a small LLM feature, three eval suites and a history of realistic pull requests
- An evaluation of false alarms and missed regressions on planted changes
If you have time left
- A history dashboard of eval scores per main-branch commit
- Budget limits per pull request, with a smaller sampled run when the full suite is too expensive
- GitLab CI support
The problem: prompts are code without tests
In a normal codebase, a pull request runs unit tests. In an LLM feature, the most important "code" is a prompt or a model name in a config file, and changing it runs no tests at all. When teams do run evals, they run them by hand, once, and compare two numbers that differ by noise.
LLM outputs vary between runs even with the same input. A suite that passes 82% today may pass 79% tomorrow without any change. Treating that as a regression wastes time; ignoring a real 5-point drop ships a worse product.
What already exists and the gap
- promptfoo runs eval suites from config files and can run in CI.
- DeepEval offers pytest-style LLM tests.
- Langfuse and LangSmith store experiment results and compare runs.
What is missing is the pull request workflow done carefully: run only what changed, compare base and head fairly, separate signal from noise with repeated runs and intervals, and present the result where the reviewer already is.
Inspiration
Software teams already use performance regression checks in CI that run benchmarks several times and compare distributions. Apply the same discipline to LLM evals, guided by the evals-first advice in "Applied LLMs: What We Learned from a Year of Building".
What you build
- A GitHub Action with change detection and a base-versus-head runner.
- Statistics that decide "regressed", "improved" or "no clear change".
- A pull request comment and a merge gate.
- A demo repository to prove it all works.
The technically interesting part
- Noise. How many repeats are enough? Measure variance on your suites and choose the smallest number of repeats that makes the decision stable.
- Cost control. Cache by prompt, model and input hash. Sample when the suite is large.
- Change mapping. A shared system prompt affects every suite; a single tool description affects one. Keep the mapping in a simple, reviewable file.
How you will evaluate it
- Plant 20 changes in the demo repo, some harmful, some neutral, some helpful, and measure how the action classifies them.
- Cost and run time per pull request with and without caching.
- A short review session with two or three developers on whether the comment helps them decide.
Path to a product
Every team with an LLM feature needs this, and it fits a hosted GitHub App with a free tier for open source. It is also a strong portfolio project for MLOps and AI platform roles.
Scope warning
Use your own demo repository and API keys with spending limits. Build on promptfoo or DeepEval for the eval engine; your contribution is the pull request workflow and the statistics.
Learn what you need on Sefism
- AI evals cheat sheet: suites, graders, variance and when a difference is real.
- MLOps roadmap: testing and release gates for models and prompts.
- GitHub Actions documentation: building and publishing the action.
- promptfoo: the eval engine you can run underneath.
- Langfuse documentation: datasets and experiment comparison.
- How to prove an AI agent project works: why evals are the proof employers look for.
- What does an AI evals engineer do?: the role that owns eval suites like these.
Get help with this idea
The risk is reporting noise as regressions until nobody trusts the bot. Tauseef can help you set up the statistics, choose a demo feature and plan the evaluation. If you are new here, create a free account first.
Ideas and guidance, not finished projects
These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.
Not sure this one fits you?
I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
Follow along
New writing, resources and project ideas land here first.