Multi-Model LLM Serving With Scale to Zero on a Few Shared GPUs

The problem: many models, few GPUs

University labs and small AI companies increasingly fine-tune open models for specific jobs: Urdu summarisation, legal document extraction, a support assistant per client. Each model or adapter is used occasionally, but keeping each one loaded on its own GPU is wasteful and, in Pakistan where GPUs are scarce and expensive, simply impossible. The usual compromise is to load models by hand when someone needs them, which takes minutes, or to stop offering them.

Serverless GPU platforms solve this in the cloud for a fee, with cold starts that can still take a long time for large models. A lab that owns a few GPUs needs the same convenience on its own hardware.

What already exists and the gap

• vLLM serves models with high throughput and supports LoRA adapters, but one server instance usually serves one base model.
• Ollama loads and unloads models on demand on a single machine, aimed at desktop use.
• KServe and similar Kubernetes tools can scale model servers to zero, but cold starts include pulling large images and checkpoints.
• Hosted serverless GPU platforms offer scale to zero as a paid service.

The gap is a self-hosted layer for a few shared GPUs that treats cold start as the core problem, combining fast local checkpoint caching, adapter sharing and placement decisions, with clear measurements.

Inspiration

"ServerlessLLM: Low-Latency Serverless Inference for Large Language Models" by Yao Fu, Leyang Xue and colleagues (OSDI 2024) uses the storage and memory of GPU servers for multi-tier checkpoint caching, a loading-optimised checkpoint format, live migration and startup-aware scheduling, and reports 10 to 200 times lower latency than earlier serverless systems. "S-LoRA: Serving Thousands of Concurrent LoRA Adapters" (MLSys 2024) shows how many adapters can share one base model. This project combines a student-sized version of both.

What you build

• Researchers and developers call one API with a model name and get a response, whether or not that model was loaded a moment ago.
• Administrators see which models are hot, which are cached where, cold start times and GPU use.
• Fine-tuners register a new adapter and it becomes available without its own GPU.

The technically interesting part

• Fast loading. Measure where cold start time goes: downloading, reading from disk, copying to GPU, initialising the engine. Then cut each part: keep checkpoints on local NVMe, keep recently used models in host RAM, use safetensors with memory mapping, and keep engine processes warm.
• Placement. Decide which models to keep on which GPU, given memory limits, request rates and the cost of loading each one. This is a caching problem with very large, unequal items. Compare least recently used with a cost-aware policy.
• Adapters versus full models. Many fine-tunes are LoRA adapters. Serving them on a shared base model avoids loading full copies; route requests so adapters of the same base land together.
• Queueing under cold start. While a model loads, requests wait. Decide how long to queue, when to return a clear "warming up" response, and how to avoid evicting a model that is about to be needed.
• Memory accounting. A model's footprint is its weights plus the KV cache for active requests, which grows with context length and batch size. Track both, so the manager does not load a model that fits at rest but runs out of memory under load.

How to evaluate it

• Cold start time for small, medium and large models from each cache tier.
• Latency percentiles under replayed traffic with many models and skewed popularity.
• GPU utilisation and the number of models served per GPU.
• Comparison with a naive baseline that loads from remote storage on demand and evicts at random.

Suggested plan

• Weeks 1 to 4: serve two models with vLLM; measure every phase of cold start.
• Weeks 5 to 9: gateway, model manager and checkpoint cache tiers.
• Weeks 10 to 15: placement policies, metrics and the first evaluation.
• Second semester: LoRA adapter serving, load generator with realistic traffic, tuning and the write-up.

Who it is for, concretely

Picture a university AI lab with two GPUs and twelve student projects, each with its own fine-tuned model, or a small company serving a separate adapter per client. Most models get a few requests an hour. Today they either sit unused or block the GPUs for everyone else.

Path to a product

Universities, AI startups and companies with on-premises data rules all want to serve many models on hardware they own. A self-hosted serving layer with a support contract, or a managed service run on a customer's GPUs, is a realistic product. It is also excellent experience for MLOps and infrastructure roles.

Scope warning

Use existing inference engines; do not write CUDA kernels. Work with models small enough for your GPUs, quantised if needed. If you only have one GPU, the project still works: placement becomes eviction on one device. Live migration is a stretch goal.

Learn what you need on Sefism

• MLOps and LLMOps roadmap: serving, monitoring and scaling models.
• MLOps and LLMOps cheat sheet: serving patterns and metrics in one place.
• Machine Learning Engineering Open Book: GPUs, storage and inference performance.
• Kubernetes Documentation: scheduling GPU workloads.
• Prometheus Documentation: the metrics behind your evaluation.
• AI engineering roadmap: how applications use the models you serve.

Get help with this idea

The risk is spending the year on setup instead of measurements. Tauseef can help you profile cold start, choose a placement policy to compare and design an evaluation that shows clear gains. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session