LLM Gateway with Semantic Cache, Model Routing and Per-User Budgets

Problem

Most applications send every prompt to the strongest model because nobody has measured whether a cheaper one would do. Similar questions are paid for again and again. One enthusiastic user, or a bug in a loop, can burn a month's budget in an afternoon. Caching and routing can cut costs sharply, but without evaluation nobody knows what they did to quality.

What you build

An HTTP gateway that exposes one OpenAI-compatible chat endpoint to internal applications and forwards requests to two or more models, for example a small and a large model from a hosted provider and a local model through Ollama.

• Authentication and budgets: API keys per application and per user, token and cost accounting per request, daily and monthly limits, and either a clear error or a downgrade to a cheaper model when a limit is reached.
• Caching: exact-match caching first. Then a semantic cache: embed each prompt, look for an earlier prompt above a similarity threshold within the same scope, and return its answer.
• Model routing: a router that classifies each request as easy or hard and picks a model. Start with rules based on length and task type, then try a small classifier.
• Observability: every request logged with model, tokens, cost, latency, cache hit or miss and routing decision, and a dashboard over those logs.
• Evaluation: a fixed set of at least 200 prompts with reference answers or grading rubrics, covering the tasks your applications actually send.

Suggested architecture

• Python with FastAPI, or Node.js with TypeScript. The gateway is an ordinary web service and does not need an agent framework.
• PostgreSQL with pgvector for cache embeddings and request logs, and Redis for the exact-match cache. Keep dependencies small and write the provider adapters yourself.
• Stream responses through the gateway and count tokens as they pass.
• Run evaluation as a script that sends the whole set through three configurations: always the large model, the router, and the router with the cache. Score with rubric grading by a strong model, and check a sample by hand.

Milestones by month

1. Months 1 and 2: gateway with two providers, API keys, streaming, request logging, first version of the evaluation set.
2. Month 3: token and cost accounting, per-user budgets, dashboard.
3. Month 4: exact-match cache, then the semantic cache with a tunable threshold, and its evaluation.
4. Months 5 and 6: rule-based router, then a learned router, both evaluated against the always-large baseline.
5. Month 7: load test, and handling a provider that is slow or down.
6. Month 8: final evaluation, report and demo.

How it will be assessed

• A results table: cost per 1,000 requests, p95 latency and quality score for each configuration, with the size of any quality drop and how confident you are in it.
• A cache analysis: hit rate at several similarity thresholds, and the number of wrong answers served from the cache at each.
• Agreement between the automated grader and human judgement on a sample of at least 50 answers.
• A demo where a user reaches their budget and a provider outage triggers a fallback.

Pitfalls

• A semantic cache that treats "fee deadline for BS Computer Science" and "fee deadline for BS Software Engineering" as the same question. Tune the threshold on real examples, and never share cache entries across users when the answer depends on the user.
• Building the router before the evaluation set. Without the set, you cannot tell whether routing helps.
• Trusting an automated grader without checking it against people.
• Logging prompts that contain personal data without redaction or a retention limit.