Cheat sheet · Specialise
Running LLMs locally cheat sheet
Opens with creditsA quick-revision book on running open-weight language models on your own machine, with Ollama as the main tool. It covers choosing and sizing models, fitting them in memory, the REST API and client libraries, local RAG, and serving safely, with a short look at llama.cpp, LM Studio and vLLM.
Curated by Tauseef Fayyaz9 chapters69 concepts89 glossary termsChapters
9 chapters · 69 concepts
1Getting started with OllamaWhat Ollama is, how to install it, and the handful of commands that cover everyday use.8 concepts
- 1What Ollama isAn open-source tool that downloads open-weight models, runs them on your CPU or GPU, and serves them through a CLI and a local HTTP API on port 11434.
- 2InstallingDesktop apps for macOS and Windows, a one-line install script for Linux that sets up a systemd service, and an official Docker image.
- 3ollama runDownloads the model if needed, loads it and opens an interactive chat, or answers a single prompt passed as an argument.
- 4Chat session commandsInside
ollama run, slash commands change settings for the session:/set parameter num_ctx 8192,/set system "...",/show info,/save name,/clear… - 5pull, list, rm and show
pulldownloads or updates a model,listshows downloaded models and their sizes,rmdeletes one, andshowprints its architecture, parameters, context… - 6ps, stop and keep-alive
ollama pslists loaded models, their memory use and whether they run on GPU, CPU or both. - 7The server and OLLAMA_HOST
ollama serveruns the server, normally started for you by the app or service. - 8Where models are storedModels are stored as content-addressed blobs under
~/.ollama/models, or in the service user's home on Linux.
8 in this chapterOpen this chapter2Models, tags and sizesHow model names and tags work, what the size in a tag means, and which kind of model fits which job.8 concepts
- 1Names and tagsA model reference is
name:tag, such asqwen3:8borgemma3:12b-it-q8_0. - 2Parameter countThe number in a tag, such as
8b, is billions of weights. - 3Instruct and base modelsInstruct (chat) models are fine-tuned to follow instructions in a conversation.
- 4Model familiesPopular open-weight families include Llama, Qwen, Gemma, Mistral, Phi, DeepSeek and gpt-oss, each in several sizes, with specialised coder, vision and embedding…
- 5Thinking modelsModels such as Qwen3, DeepSeek-R1 and gpt-oss can reason step by step before answering.
- 6Vision modelsModels such as Gemma 3 and Qwen-VL variants accept images alongside text, through the
imagesfield in the API or a file path in the CLI prompt. - 7Embedding modelsSmall models such as
nomic-embed-textormxbai-embed-largethat turn text into vectors instead of generating text. - 8Models from Hugging FaceAny GGUF repository on Hugging Face can be run directly with
hf.co/user/repo, optionally with a quantisation tag.
8 in this chapterOpen this chapter- 1Names and tagsA model reference is
- Quantisation and memoryHow weights are compressed, what the quantisation names mean, and how to estimate whether a model and its context will fit.7 conceptsOpens with credits
- Hardware and performanceWhere the model runs, why memory bandwidth sets the speed, and what happens when a model does not fit.8 conceptsOpens with credits
- Modelfiles and context lengthPackaging a model with its parameters, system prompt and template, and setting the context window it actually uses.8 conceptsOpens with credits
- The local REST APIEvery Ollama feature is available over HTTP on port 11434: native endpoints for generation, chat and embeddings, plus an OpenAI-compatible layer.8 conceptsOpens with credits
- Clients, structured outputs and toolsThe official Python and JavaScript libraries, and the features that turn a local model into an app component: JSON schemas, tool calls and thinking.7 conceptsOpens with credits
- Embeddings and local RAGBuilding retrieval-augmented generation entirely on your machine: embedding documents, storing vectors and answering from them.7 conceptsOpens with credits
- Serving safely and alternativesExposing Ollama beyond localhost without exposing yourself, running it in Docker, and when llama.cpp, LM Studio or vLLM fit better.8 conceptsOpens with credits
- 10Glossary, A to ZEvery concept in the sheet and the smaller terms it uses, in one list.89 terms
About this sheet
Ollama commands, Modelfiles, quantisation, memory and hardware, the local API, clients, tools, embeddings and safe serving.
Related on Sefism
Picked from the other libraries by shared topics.
Roadmaps
See allBest practices
See all- Best Practices for Building LLM AppsHow to build LLM apps and agents that hold up in production: prompts, evals, prompt injection, tool permissions, cost, logging and privacy.
- Prompt Engineering Best PracticesHow to write prompts that work the same way on the hundredth input as on the first: clear instructions, structure, examples, reasoning, output control and testing.
- Best Practices for AI Coding AssistantsHow to get reliable code from AI coding assistants and agents: instruction files, task prompts, verification, review, security, dependencies and session hygiene.
Checklists
See all- RAG Production ChecklistA RAG production checklist: chunking, embeddings, hybrid search, reranking, per-user permissions, citations, evals, re-indexing and cost, with how to verify each check.
- LLM Evaluation ChecklistAn LLM evaluation checklist: success criteria, eval datasets, code and model graders, judge calibration, agent evals, safety tests and production monitoring, with how to verify each check.
- AI Feature Launch ChecklistAn AI feature launch checklist: use case fit, evals, UX for errors, disclosure, human override, abuse filters, rate limits, cost caps, prompt privacy, monitoring and rollback.
Project structures
See all- GenAI RAG app (FastAPI) project structureFor developers turning a "chat with your documents" demo into an app people can rely on. A FastAPI backend where every layer has a home: ingestion workers, retrieval, prompts as versioned files, model clients with fallbacks and cost caps, guards, evals that gate CI, and tracing, so a wrong answer can be traced to the step that caused it.
- AI agent with an MCP server project structureFor developers who want to give AI assistants safe access to their own data or API. A TypeScript MCP server built on the official SDK, organised by feature (tasks, projects) with each feature owning its tools, resources and prompts, served over stdio for local hosts and Streamable HTTP with OAuth for remote ones, plus a small agent and the evals that prove a model can use it.
- Machine learning project project structureFor students and analysts whose model works in a notebook and now needs to be reproducible, reviewable, served and watched. It starts from the Cookiecutter Data Science v2 layout (immutable raw data, numbered notebooks, a real Python package) and grows into versioned data with DVC, typed parameters, data validation with pandera, experiment tracking and a model registry with MLflow, a FastAPI prediction service, drift monitoring and, for a team, scheduled retraining, model cards, fairness checks and shadow deployment.
Career questions
See all- Which AI tools do employers expect junior developers to know in 2026?Employers rarely require a specific brand. They expect you to use an AI coding assistant well (in the editor and in the terminal), to review and test what it produces, to call an LLM API in a small feature, and to explain when you would not use AI. Pick one assistant, learn it properly, and show judgement in your projects.
- How do I use AI to become a better engineer instead of becoming dependent on it?Decide the job AI does before you open it. Use it as a tutor, reviewer and sparring partner that explains and challenges, not a vending machine that hands you finished code. Attempt first, ask second, and always be able to explain and defend what you ship.
- How do I start learning AI agents as a software engineer?Start with one LLM API and plain code: a loop that calls the model, lets it pick a tool, runs the tool and feeds the result back. Build one small agent that does a real task end to end, add evaluation and guardrails, and only then reach for a framework. Agents are software engineering first.
Work with me
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
- 1:1 Mentorship
- Fresh Graduate Mentorship
- Career Guidance
Brand deals, promos and partnerships: tauseeffayyaz@gmail.com