Learned LLM Router That Sends Easy Tasks to Cheap Models, with a Cost-Quality Frontier
Many LLM requests are easy enough for a small, cheap model, and a few need a strong, expensive one. Build a router that learns from your own evaluation data which requests need which model, compare it with simple rules and with always using one model, and plot the cost-quality frontier so a team can pick an operating point. Include cost tracking per request and a fallback when the cheap answer looks weak.
Advanced
Needs a team, real planning, and a supervisor who knows the area.2 semesters · 2 students
6 to ship
3 optional extrasSuggested stack
What you should ship
- A task mix of at least 2,000 requests from public datasets across categories such as classification, extraction, summarisation and reasoning, plus Roman Urdu items
- Answers from at least three models of different price and strength, graded automatically or by an LLM judge checked against human labels
- Routers to compare: rule-based by task type, a classifier on prompt embeddings, and a cascade that tries the cheap model first and escalates on low confidence
- A cost-quality frontier plot for each router, with the always-cheap and always-strong baselines
- A router service with an OpenAI-compatible API, per-request cost logging and a budget setting
- A held-out test showing how the router generalises to a new category it was not trained on
If you have time left
- Online learning from user thumbs-up and thumbs-down
- Including a local open model as the cheapest option
- Latency as a third axis
The problem: paying frontier prices for easy questions
Teams often pick the strongest model and send it everything, because it is safe. Then the bill arrives. Other teams pick the cheapest model and lose quality on the hard requests. Neither is optimal: in most products, a large share of requests are easy.
The question is how to tell, before answering, which request needs which model, and how much quality you give up for each unit of savings.
What already exists and the gap
- FrugalGPT (Chen, Zaharia and Zou, 2023) proposed cascades and showed large savings on several tasks.
- RouteLLM (from the LMSYS team, published at ICLR 2025) trained routers on preference data and reported large cost reductions on standard benchmarks while keeping most of the strong model's quality.
- Gateways such as LiteLLM and OpenRouter make multi-model routing easy to wire up, and commercial routers such as Not Diamond and Martian sell learned routing.
The gap for an FYP is a careful, reproducible comparison on a task mix that includes local-language requests, with a frontier plot a team can use to choose, and a small router service that runs in production.
Inspiration
RouteLLM's insight is that the router itself can be cheap: a small classifier on embeddings can predict whether the weak model's answer will be good enough. Your project tests that idea on a different task mix and asks whether it transfers.
What you build
- A dataset of graded answers from several models.
- Three routers and two baselines.
- Frontier plots and a router service with cost tracking.
The technically interesting part
- Labels are expensive. Every training example needs answers from several models and a grade. Plan a budget and use caching.
- Cascades versus predictors. A cascade pays for the cheap answer every time; a predictor does not, but can be wrong. Compare honestly.
- Generalisation. Routers can overfit to task categories. A held-out category is the real test.
How you will evaluate it
- Quality at matched cost and cost at matched quality, for each router.
- Frontier plots with confidence bands.
- Router overhead in milliseconds and cost.
Path to a product
Every company with meaningful AI spend wants lower bills without worse answers. A self-hosted router with a dashboard and cost reports, or routing as a feature in an AI gateway, are proven product shapes.
Scope warning
Set hard spending limits on API keys. Do not build a gateway from scratch; use LiteLLM and focus on the router and the evaluation. Record model versions, because providers update models.
Learn what you need on Sefism
- Choosing LLMs roadmap: comparing models on cost, quality and latency.
- Choosing LLMs cheat sheet: a one-page view of the trade-offs.
- FinOps roadmap: unit costs and showing savings in a way finance trusts.
- LiteLLM documentation: calling many models through one interface with cost tracking.
- scikit-learn user guide: the classifier behind the router.
- AI evals cheat sheet: grading answers reliably.
- How do I choose which LLM to use for my project?: the single-model decision your router automates.
Get help with this idea
The risk is running out of API budget before you have enough labelled data. Tauseef can help you plan the dataset, the budget and a comparison that holds up. If you are new here, create a free account first.
Ideas and guidance, not finished projects
These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.
Not sure this one fits you?
I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
Follow along
New writing, resources and project ideas land here first.