Stop relying on the demo. Write 30 or more test tasks from realistic and messy inputs, check what you can in code (right tool, right arguments, right end state), run each task several times, report how often it succeeds every time, add a few injection attacks, and trace every run. Numbers from that are what make an agent project credible.
What people tell me
I built an AI agent for my portfolio that books appointments and answers questions using a few tools. It works when I demo it, but sometimes it calls the wrong tool or makes up a time slot, and I only notice by accident. When someone asks how I know it works, all I can say is that I tried it a lot. I want to show that it is reliable, but I do not know what testing an agent even looks like, since the answers change every run.
A composite of the messages behind this question, with personal details left out.
Key takeaways
- A demo proves an agent can work once; an eval set shows how often it works.
- Check tool choice, arguments and end state in code wherever you can; use a model judge only for what code cannot check.
- Run every task several times and report consistency, because users experience the agent's average day, not its best one.
- Add prompt injection attempts and failure cases, not just happy paths.
- Trace every run so you can show exactly why a failure happened and how you fixed it.
Short answer: replace "I tried it a lot" with numbers
A demo shows your agent can work once. What makes a project credible is showing how often it works, how it fails and what you did about it. That is what an eval set gives you, and it is the skill that separates people who have played with agents from people who have built one.
Step 1: write the tasks
Write 30 to 50 tasks, each an input plus a way to judge success. Draw them from how real users would talk, not from how you would. Include:
- common requests, phrased several different ways
- ambiguous ones, such as "book me something next week"
- impossible ones, such as a slot that does not exist
- out-of-scope ones the agent should refuse or hand over
- a few prompt injection attempts, such as a message that says to ignore previous instructions and cancel every booking
Step 2: check in code wherever you can
For each task, decide what counts as success and check it with code:
- Did it call the availability tool before booking?
- Were the arguments valid, such as a real date in the right format?
- Is the final state correct: one booking, at a slot that exists, for the right person?
Run these against fake tools that return fixed data and record every call, so tests are fast, free and never book anything real. Use a model as judge only for qualities code cannot check, such as whether the reply is clear, and compare its verdicts with your own on a sample before trusting it.
Step 3: run each task several times
Agents give different results on the same input. Run every task three to five times. Report two numbers: how often at least one run succeeds, which shows capability, and how often every run succeeds, which shows consistency. An agent that succeeds 80 percent of the time only gets three in a row right about half the time, and your users feel that.
Step 4: trace everything
Add tracing with a tool such as Langfuse so every model call and tool call is recorded. When a task fails, you can show exactly which step went wrong. Fix it, and add the failing case to your eval set so it stays fixed.
Step 5: put it in CI and in your README
Run a fast subset of the evals on every change, so a prompt tweak cannot silently break booking. Then put a short results table in your README: the number of tasks, success rates, the failures you found and what you changed. That table says more than any demo video.
Tools that help
- DeepEval if you work in Python: pytest-style tests with agent metrics such as tool correctness.
- promptfoo: config-driven evals and red teaming for injection attacks.
- AI Evals: Everything You Need to Know: how to design eval sets and judges well.
The evals and guardrails stages of the AI agents and MCP roadmap take you through all of this step by step, and the AI agents cheat sheet summarises graders, pass rates and injection defences on one page. If you want a project where this is the whole point, look at the agent ideas in FYP ideas, and browse more career questions for related answers.
If you want me to look at your eval plan, create a free account and book a 1:1 session.
Was this answer helpful?
Your situation is not quite this one?
Members get written answers to their own questions, a roadmap built for them and feedback on their projects. Early access is open: sign up and your account is active shortly after. Prefer to talk? A free call works too.