How LLM apps and agents break

Follow a request through an LLM app and an agent to see where text gets in and where actions get out.

Also called context window, system prompt, LLM app anatomy

What: The text an app builds for the model: system prompt, history, retrieved documents, tool descriptions and results, and the user's message.
Why: Every source feeding the context is a way in; every place the answer goes is a way out.

Watch out

Treating the chat box as the only input.

Also called model is not a security boundary, untrusted output

What: A model reads instructions and data as the same stream of text and decides by probability which parts to follow.
Why: It is the root cause of prompt injection, so rules must be enforced in code outside the model.
Example: "Never reveal customer data" in a prompt is a hope; a query that returns only the user's rows is a control.

Watch out

Believing a stronger model or a cleverer system prompt removes the problem.

Also called RAG, vector store, embeddings, tenant isolation

What: Filtering retrieval by who is asking, in the query itself, and controlling who can write to indexed sources.
Why: Without it the model becomes a way to read any document in the store, including other customers' files.

Watch out

Filtering after the model has already read the chunks.

Also called tool calling, function calling, AI agent

What: A loop where the model asks for a tool call, code runs it, the result returns to the context, and the model decides again.
Why: Tools turn a wrong answer into a wrong action, and each loop step can bring in new untrusted text.

Watch out

Reviewing the first prompt and ignoring what enters on later steps.

Also called probabilistic behaviour, sampling

What: The same input can give different outputs across runs and model versions.
Why: Defences and failures must be measured as rates and retested after every change.

Watch out

Reporting "the defence held" after a handful of attempts.

Also called private data, untrusted content, external communication

What: Simon Willison's name for three capabilities that together let an injection steal data: private data, untrusted content and a way to communicate out.
Why: Removing any one leg blocks this whole class of attack, which makes it a fast design check.

Watch out

Adding a web fetcher or image rendering to an agent that already reads private data.

Also called home lab, local model

What: A small LLM app you own: a local or spend-capped model, a system prompt, retrieval over your own documents and two harmless tools.
Why: You can see every prompt and tool call, test cheaply and stay within the law.

Watch out

Practising on a company's live chatbot.

Go deeper