Running LLMs locally

Sefism.Running LLMs locally

Tauseef Fayyaz

A to Z

Glossary

Every concept in this sheet, plus the smaller terms it uses. Select a concept to open its card.

C

Chat session commands: Inside ollama run, slash commands change settings for the session: /set parameter num_ctx 8192, /set system "...", /show info, /save name, /clear and /bye.

E

Embedding models: Small models such as nomic-embed-text or mxbai-embed-large that turn text into vectors instead of generating text.

I

Installing: Desktop apps for macOS and Windows, a one-line install script for Linux that sets up a systemd service, and an official Docker image.
Instruct and base models: Instruct (chat) models are fine-tuned to follow instructions in a conversation.

M

Model families: Popular open-weight families include Llama, Qwen, Gemma, Mistral, Phi, DeepSeek and gpt-oss, each in several sizes, with specialised coder, vision and embedding variants.
Models from Hugging Face: Any GGUF repository on Hugging Face can be run directly with hf.co/user/repo, optionally with a quantisation tag.

N

Names and tags: A model reference is name:tag, such as qwen3:8b or gemma3:12b-it-q8_0.

O

ollama run: Downloads the model if needed, loads it and opens an interactive chat, or answers a single prompt passed as an argument.

P

Parameter count: The number in a tag, such as 8b, is billions of weights.
ps, stop and keep-alive: ollama ps lists loaded models, their memory use and whether they run on GPU, CPU or both.
pull, list, rm and show: pull downloads or updates a model, list shows downloaded models and their sizes, rm deletes one, and show prints its architecture, parameters, context length, template and licence.

T

The server and OLLAMA_HOST: ollama serve runs the server, normally started for you by the app or service.
Thinking models: Models such as Qwen3, DeepSeek-R1 and gpt-oss can reason step by step before answering.

V

Vision models: Models such as Gemma 3 and Qwen-VL variants accept images alongside text, through the images field in the API or a file path in the CLI prompt.

W

What Ollama is: An open-source tool that downloads open-weight models, runs them on your CPU or GPU, and serves them through a CLI and a local HTTP API on port 11434.
Where models are stored: Models are stored as content-addressed blobs under ~/.ollama/models, or in the service user's home on Linux.