vLLM Documentation

What vLLM is

vLLM is a fast, open source library for LLM inference and serving. It was originally developed in the Sky Computing Lab at UC Berkeley and is now built by a large community of universities and companies. If you want to self-host an open-weight model for many users, rather than call a paid API, vLLM is the default place to start. Hugging Face put its own TGI server into maintenance mode in December 2025 and points new deployments to vLLM or SGLang.

The ideas that make it fast

• PagedAttention: the KV cache, the attention data kept for every token already processed, is stored in small blocks, so far less GPU memory is wasted and more requests fit at once.
• Continuous batching: new requests join the running batch between generation steps instead of waiting for a whole batch to finish.
• Prefix caching and chunked prefill: shared prompt prefixes are reused, and long prompts are split so they do not stall other requests.
• Quantisation: support for formats including FP8, INT8, INT4, GPTQ, AWQ and GGUF, so larger models fit on smaller GPUs.

Serving in practice

vllm serve <model> starts an OpenAI-compatible API server, and recent versions also offer an Anthropic Messages API and gRPC. Existing OpenAI client code works by changing the base URL. The settings you will tune first are the maximum context length and the share of GPU memory vLLM may use, which together decide how many concurrent requests fit. The docs cover NVIDIA and AMD GPUs, CPUs and plugins for other accelerators such as Google TPUs and Intel Gaudi.

Prerequisites

Comfort with Python, the command line and Docker, and access to a GPU: a free notebook GPU is enough to experiment with small models. Understanding tokens, context windows and the KV cache makes the configuration pages much easier.

How long it takes

Serving a small model and calling it takes an afternoon. Tuning memory, quantisation and concurrency with a proper load test is a weekend.

Free or paid?

vLLM is free and open source under the Apache 2.0 licence. You pay only for the hardware it runs on.

Common questions

vLLM or Ollama? Ollama is excellent for running a model on your own laptop. vLLM is built for serving many concurrent users on server GPUs.

What can I do after it? Serve an open or fine-tuned model behind an OpenAI-compatible API, size the GPU for your traffic, and compare its cost per million tokens with a hosted API.

Best for: engineers moving from LLM APIs to self-hosted open models, and anyone building the serving part of an LLMOps project.