Learn classic system design first, because AI applications are still web systems with queues, caches and databases. Then add what is new: slow and costly model calls, retrieval pipelines, vector search, evaluation, guardrails and cost control. Practise by designing and building one AI feature end to end.
What people tell me
I understand basic system design for normal web apps: load balancers, caches, databases. Now most new projects at work and in job posts involve LLMs, RAG or agents, and I do not know how to design them properly. Tutorials show a single API call in a notebook, but nothing about how these systems work in production with real users, costs and failures.
A composite of the messages behind this question, with personal details left out.
Key takeaways
- AI applications are ordinary distributed systems with one slow, expensive and non-deterministic component.
- Master the classic building blocks first: APIs, queues, caches, databases and observability.
- Then learn the AI-specific parts: retrieval, vector search, streaming, evaluation, guardrails and cost.
- Design for failure: timeouts, fallbacks, retries and a human in the loop for risky actions.
- Build one small AI feature end to end and write a design document for it.
Start with what has not changed
An AI application is still a web application. It has users, an API, a database, background jobs, caching, authentication and monitoring. The model is one component inside it: an unusually slow, costly and unpredictable one. If you skip the classic foundations, every AI design you draw will be missing half the system.
So the first step is ordinary system design: how requests flow, how data is stored, how to scale reads and writes, how queues decouple slow work, how caches trade freshness for speed. The System design roadmap and cheat sheet cover these, and The System Design Primer is a good free reference.
What is different about AI systems
| Property of the model | Design consequence |
|---|---|
| Slow (seconds, not milliseconds) | Stream responses; move long work to background jobs with status updates |
| Costs money per token | Cache results, trim context, route easy requests to smaller models, set budgets per user |
| Non-deterministic | Validate structured output, retry with limits, keep a fallback path |
| Only knows its training data | Add retrieval so it can use your own, current data |
| Can be manipulated by input | Treat retrieved content and user text as data; guard tools and actions |
| Hard to test with simple assertions | Build an evaluation set and score outputs over time |
The building blocks to learn
Retrieval (RAG). Most business AI features need the model to answer from your documents or data. Learn the pipeline: chunking documents, creating embeddings, storing them in a vector index, retrieving the best matches, and passing them to the model with the question. Then learn why it fails: bad chunking, stale indexes, retrieval that finds similar but wrong text. RAG Techniques and the Retrieval Augmented Generation course are good starting points.
Model gateway. A single internal service that all model calls go through, handling keys, retries, timeouts, logging, cost tracking and switching providers. Tools such as LiteLLM show what this layer looks like.
Agents and tools. When the model can call functions, you need step limits, permission checks and approval for anything destructive. Building Effective Agents explains when an agent is worth the extra complexity and when a fixed workflow is better.
Evaluation and observability. Log every prompt, response, retrieved document and cost. Keep a test set of real questions and rerun it after every change. AI Evals: Everything You Need to Know is a practical introduction.
Safety. Prompt injection, data leakage and unsafe outputs are design problems, not afterthoughts. The OWASP Top 10 for LLM Applications is the checklist to design against.
Practise with real designs
Pick one system a week and design it on paper, then compare with how real teams describe similar systems:
- A support assistant that answers from a help centre and hands off to a human.
- A document question-answering tool for a company's internal policies, with access control per department.
- A code review bot that comments on pull requests.
- A meeting summariser that processes long recordings in the background.
For each, answer: where does data come from, how fresh must it be, what happens when the model is slow or down, how much does one request cost, how will you know answers are good, and what must never happen.
Build one end to end
Designs on paper only go so far. Build a small version of one of those systems with a real API, background processing, a vector index, logging and an evaluation script. Then write a two-page design document explaining the choices and trade-offs. Designing Machine Learning Systems and Designing Data-Intensive Applications are the books to read alongside.
The AI engineering roadmap puts all of this in order, from prompting to retrieval to agents and evaluation.
If you are still stuck
If classic system design still feels shaky, start with How do I learn system design as a student or junior developer?. If you want feedback on a design you are working on, join Sefism and, as a member, book a 1:1 session.
Was this answer helpful?
Read next
- A Learning Roadmap for System DesignA system design roadmap that fixes the order: foundations, vocabulary, data, services talking to each other, then real systems. How long each stage takes, how to know you are done, and the free and paid resources worth using.
- RAG, Fine Tuning, or a Better Prompt: How to ChooseRAG vs fine tuning vs a better prompt: a practical order from prompting through retrieval, embeddings, reranking, tools and agents to fine tuning, with the signal that tells you to move to the next step.
- An AI Engineering Roadmap That Puts Things in the Right OrderAn AI engineering roadmap in seven stages, from tokens and prompting through RAG, evals, tool calling, MCP and agents, with a project to build at each step, what to skip, and a realistic timeline.
Your situation is not quite this one?
Members get written answers to their own questions, a roadmap built for them and feedback on their projects. Early access is open: sign up and your account is active shortly after. Prefer to talk? A free call works too.