Evaluation Harness for Language Model Applications

The problem teams have when changing prompts

Teams building on language models change a prompt, feel that the output improved, and ship it. There is no baseline, no test set and no way to know whether the change fixed one case and broke five. The same happens when switching to a cheaper model or a new version: someone tries a few examples by hand and decides. It is one of the most common gaps in applied AI work, and it gets expensive once a feature has real users.

Software engineering solved this for ordinary code with unit tests and CI. Model outputs are open-ended and non-deterministic, so the same idea needs different tools.

What already exists and the gap

• promptfoo is an open-source command line tool and library for testing and red-teaming model applications, now part of OpenAI and still MIT licensed.
• OpenAI Evals is an open framework and registry for evaluating models.
• DeepEval is a pytest-style open-source evaluation library.
• Langfuse and Braintrust combine tracing with datasets and evaluation.

This space is active, and you should learn from these tools. The gap a student project can own is depth on two things they leave to the user: how trustworthy the model grader is for a given task, and how flaky results are across repeated runs. A harness that reports grader agreement and run-to-run variance alongside every score gives teams numbers they can actually act on.

What you build

Users: a developer who writes suites and reviews runs, and a CI pipeline that blocks regressions.

• Suite format and runner.
• Result store with cost and latency.
• Comparison view focused on flipped cases.
• CLI and GitHub Actions integration.
• Grader calibration workflow: humans label a sample, the harness reports agreement.

The technically interesting part

The hard problem is model-graded assertions, where one model judges another's output. They are necessary for anything open-ended, and they are unreliable in ways that are easy to miss: position bias, preferring longer answers, leniency toward their own style. Build the calibration loop into the product: sample outputs, collect human labels, compute agreement (Cohen's kappa or simple accuracy), and show it next to every rubric score. If agreement is low, the harness should say so.

The second is non-determinism. A case that passes 7 times out of 10 is not a pass. Repeated runs and a clear flakiness metric keep teams from chasing noise.

How to evaluate it

• Case studies: use the harness on two real tasks and report findings. A conclusion such as "prompt B is eight percent more accurate and forty percent more expensive" is what this project should produce.
• Grader agreement with human labels on at least 100 outputs per task, before and after rubric tuning.
• Regression catch: seed 10 known regressions into prompt versions and check how many the CI gate catches.
• Flakiness: variance of pass rate across 5 repeated runs.
• Developer test: three developers write a suite for a new task and report time taken and friction.

Suggested plan

1. Weeks 1 to 4: suite format, deterministic assertions, runner, result store.
2. Weeks 5 to 9: comparison view and CLI.
3. Weeks 10 to 14: first worked suite, CI integration.
4. Semester 2, weeks 1 to 5: model-graded assertions and calibration workflow.
5. Weeks 6 to 10: repeated runs, flakiness, second worked suite.
6. Weeks 11 to 14: regression experiment, developer test, report.

Path to a product

The first users are small AI teams at software houses delivering chatbots and extraction features to clients, who need evidence that a change did not break the client's use case. Release the harness as open source; charge for a hosted dashboard with history and team sharing, or offer evaluation setup as a consulting service. Realistically, the strongest outcome may be the portfolio: evaluation is one of the most in-demand AI engineering skills.

Scope warning

• Do not build a full experiment tracking platform.
• Do not build authentication, teams or billing.
• Do not support every model provider; two is enough.
• Do not trust any model grader number you have not checked against humans.

Learn what you need on Sefism

• AI engineering roadmap : evaluation, prompting and model APIs.
• DevOps and cloud cheat sheet : CI pipelines for the regression gate.
• LiteLLM Documentation : one interface to call several providers from your runner.
• JSON Schema: Getting Started : schema assertions for structured outputs.
• How to use AI to become a better engineer : the mindset behind measuring rather than guessing.

Get help with this idea

The part examiners will value most is the grader calibration and the regression experiment, not the number of assertion types. Tauseef can help you scope the harness, pick realistic example tasks and design the human labelling. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session