Fine-Tuned Small Open Model with Eval-Gated Releases and Self-Hosted Serving

The problem

Many startups and support teams use a large hosted model for one narrow job repeated thousands of times a day: tagging support tickets and pulling out order numbers, classifying delivery messages, or turning free text into a fixed JSON form. The task is stable, the format is fixed, and the bill grows with every customer. A small open model fine-tuned on that one task can often match the large model on it, run on a single GPU the team controls, and keep customer text in-house. But teams rarely have the evidence or the pipeline to switch safely.

This project builds that pipeline: fine-tune, evaluate against the API baseline, release only through an eval gate, serve, trace, and report whether switching is worth it.

What exists and the gap

Hosted fine-tuning is offered by model providers, such as OpenAI's supervised fine-tuning API, and by inference platforms such as Together AI. Open tools such as Unsloth, Hugging Face TRL and PEFT make LoRA fine-tuning of open models practical on modest hardware, and vLLM serves the result efficiently. What is missing for most small teams is the release discipline around them: a versioned eval set, a gate that compares every candidate with production, tracing and an honest cost comparison. Without those, a fine-tuned model is a gamble.

Inspiration

• LoRA: Low-Rank Adaptation of Large Language Models (Hu and colleagues, Microsoft, 2021) introduced the adapter approach this project uses.
• QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers and colleagues, University of Washington, 2023) showed that fine-tuning a 4-bit base model with adapters can preserve quality while fitting on a single GPU.

Scope for a final year project

Pick one task and one real or realistic dataset: public customer support or e-commerce message datasets work, or anonymised messages from a partner company, in English, Urdu or Roman Urdu. Keep the task narrow and measurable, such as a category plus three extracted fields as JSON.

1. Build the eval set first and score the hosted API baseline on it.
2. Prepare and version the training data; fine-tune a small open model with QLoRA on a free or low-cost GPU.
3. Put the eval gate in CI so no model or prompt change ships if it regresses.
4. Serve the winner with vLLM, put a router and fallback in front, and trace every request.
5. Load test, measure, and write the comparison.

What to measure

Exact match and field-level F1 on the held-out set, valid JSON rate, results by message type and language, p95 latency, throughput per GPU, and cost per 1,000 requests for both options, using your own measured throughput and the providers' published prices at the time you write the report.

Path to a product

The reusable part is the pipeline, not one model. Package it as a service that takes a team's task definition and examples and returns a gated, served, monitored model with a cost report, starting with software houses that run the same kind of task for many clients. Data residency, keeping customer text inside the company or the country, is a strong selling point for banks, telecoms and health providers.

Discuss it and get help

Want to shape this idea for your team? Discuss it at /account/fyp-ideas/fine-tuned-small-open-model-with-eval-gated-releases/, ask a question at /account/ask/, or book time with Tauseef at /account/book/. New to Sefism? Create a free account.

Learn what you need on Sefism

• MLOps and LLMOps roadmap: the LLMOps and fine-tuning stages map directly onto this project
• MLOps cheat sheet and MLflow cheat sheet
• Unsloth documentation for LoRA and QLoRA fine-tuning notebooks
• vLLM documentation for serving open models
• promptfoo and AI Evals: Everything You Need to Know for the eval gate
• Langfuse documentation for tracing and AI Engineering for the bigger picture