Tool Selection Evals and Description Linter for MCP Servers
An MCP server can work perfectly and still fail in practice because models pick the wrong tool or fill its arguments badly. Build a tool that lints MCP tool names, descriptions and schemas against good-practice rules, generates realistic user requests for each tool, measures how often several models choose the right tool with valid arguments, and suggests rewrites that improve the score.
Intermediate
A small team, or one strong student willing to learn something new.1 semester · 2 students
7 to ship
3 optional extrasSuggested stack
What you should ship
- A connector that starts any MCP server, lists its tools and schemas, and records them
- A linter with documented rules: overlapping tools, vague descriptions, missing parameter descriptions, ambiguous names, overly broad tools, oversized outputs
- A request generator that writes realistic user requests per tool, including tricky near-misses that belong to a different tool
- A selection eval that measures, for three or more models, right tool chosen, arguments valid against schema, and arguments correct
- A rewrite assistant that proposes better descriptions and re-runs the eval to show the effect
- Results on at least five open source MCP servers, shared with their maintainers as suggestions
- A report on which description changes actually helped
If you have time left
- Measuring how selection accuracy drops as more servers and tools are connected at once
- A GitHub Action that runs the linter and eval on each release of a server
- Testing small open models that run on a laptop
The problem: the right tool, described badly
Agents work by choosing tools from descriptions. If two tools sound alike, if a parameter has no description, or if a tool returns a wall of text, models make mistakes even when the server's code is perfect. As people connect more MCP servers, the list of tools a model sees grows, and confusion grows with it.
Anthropic's engineering article "Writing effective tools for agents" recommends evaluating tools with realistic tasks and improving descriptions based on results. Most server authors never do this.
What already exists and the gap
- The Berkeley Function Calling Leaderboard (BFCL) measures how well models call functions on a fixed benchmark.
- MCP Inspector lets a developer call tools by hand.
- Security scanners check tool descriptions for injection.
The gap is a developer tool that tests a specific server's tool design with several models and turns the results into concrete description fixes. BFCL ranks models; your tool improves servers.
Inspiration
Search engines improved by measuring which results people actually clicked. Tool design can improve the same way, by measuring which tools models actually pick for realistic requests, and fixing the ones they miss.
What you build
- A linter with rules backed by evidence from your own experiments.
- A request generator and selection eval.
- A rewrite loop that proves a better description helps.
The technically interesting part
- Hard negatives. The most useful test requests are ones that sound like tool A but belong to tool B. Generate and hand-check them.
- Scaling the tool list. Accuracy may be fine with 5 tools and poor with 60. Measure the curve.
- Avoiding overfitting. If the rewrite assistant tunes descriptions to your generated requests, hold out a set written by people.
How you will evaluate it
- Selection and argument accuracy before and after rewrites, on held-out human-written requests.
- Linter precision: how many flagged issues did maintainers agree were real?
- Model comparison on the same servers.
Path to a product
Companies publishing MCP servers for their products want agents to use them correctly. A testing service or CI tool for MCP server authors is a natural niche, and contributions to popular open source servers make a strong portfolio.
Scope warning
Test servers locally with test accounts; do not point them at production data. Five servers and three models are enough.
Learn what you need on Sefism
- AI agents roadmap: how agents choose and call tools.
- MCP cheat sheet: the protocol in one page.
- AI evals cheat sheet: building and holding out test sets.
- Writing effective tools for AI agents: the guidance your linter turns into rules.
- MCP Inspector: poking at servers by hand.
- Claude tool use documentation: how a model sees tool definitions.
- OpenAI function calling guide: the same idea from another provider.
- What does an AI evals engineer do?: evaluation as a job, beyond this project.
Get help with this idea
The risk is a linter full of opinions with no evidence. Tauseef can help you design the experiments that turn rules into proven advice. If you are new here, create a free account first.
Ideas and guidance, not finished projects
These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.
Not sure this one fits you?
I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
Follow along
New writing, resources and project ideas land here first.