Evaluation Harness for Language Model Applications
A tool that lets a developer define test cases for a language model feature, run them against several models or prompt versions, and see which changes actually improved things. Most teams building with these models have no way to answer that question, which makes every decision after the first a guess.
Advanced
Needs a team, real planning, and a supervisor who knows the area.1 semester, 2 to 3 students
5 to ship
3 optional extrasSuggested stack
What you should ship
- Test case format supporting exact match, contains, regex and model graded assertions
- Runner that executes a suite against a configured model and stores every result
- Comparison view showing two runs side by side with per case pass and fail differences
- Cost and latency recorded per run so a quality gain can be weighed against what it costs
- At least two worked example suites, such as a classifier and a summariser, with results reported
If you have time left
- Regression detection that flags cases which passed in the previous run and now fail
- Command line runner suitable for a continuous integration pipeline
- Model graded evaluation with agreement measured against human labels
The problem
Teams building on language models change a prompt, feel that the output improved, and ship it. There is no baseline, no test set, and no way to know whether the change fixed one case and broke five. This is the single most common gap in applied AI work right now.
What you build
A harness where a developer writes test cases declaratively, runs the suite against a model and prompt configuration, and gets a report. The valuable part is the comparison: run two configurations and see exactly which cases changed, in which direction, and at what cost in tokens and latency.
Why it is a strong project
It is genuine infrastructure rather than a demonstration, the evaluation of your own work is concrete, and the topic is current enough that the write up has real value. It also demonstrates engineering judgement rather than model knowledge, which is easier to assess fairly.
The hard part
Model graded assertions, where one model judges another's output. They are necessary for anything open ended and they are unreliable in ways that are easy to miss. If you build them, you must measure how well the grader agrees with human judgement on a sample, and report it honestly.
How to evaluate it
Use your own tool on two real tasks and report the findings. A result such as prompt B being eight percent more accurate and forty percent more expensive is exactly the kind of conclusion this project should produce.
Scope warning
Do not build a full experiment tracking platform. Test definition, execution, comparison and cost is a complete semester. Authentication, teams and sharing are not.
Ideas and guidance, not finished projects
These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.
Not sure this one fits you?
I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Every session is free; a few slots open each week.
Follow along
New writing, resources and project ideas land here first.