AI security: secure and test LLM apps and agents

Stage 2 of 9 · 7 lessons · ~5 weeks

How LLM apps and agents work, and where they break

Follow a request through an LLM app and an agent: how the context window is assembled, why instructions and data mix, where retrieval and tools add risk, why behaviour varies run to run, and the practice app you will defend and test for the rest of the roadmap.

Start: How a request becomes a context window

Lessons· one lesson for each topic, in this order

  1. 1How a request becomes a context window
  2. 2Instructions and data share one channel
  3. 3Where retrieval goes wrong
  4. 4The agent loop as an attack surface
  5. 5Probabilistic behaviour and what it means for security
  6. 6Private data, untrusted content, a way out
  7. 7Build the app you will defend and test

Do this

  • Run a small model locally or set a spending limit on a hosted API.
  • Build a chat app with a system prompt and retrieval over a few documents you own.
  • Add two harmless tools and write down what each can do.
  • Draw the context assembly with every source and destination.
  • Mark the trifecta legs for your app and note which tool gives each.
  • Run the same risky request 20 times and record how often the answer changes.

Start with· follow one of these

  • CoursePaid
    Grokking Prompt Engineering for Software Engineers

    Design Gurus · Beginner · Subscription

    How system prompts, context and structured output really work, the ground every LLM risk stands on

  • CourseFree
    Hugging Face Learn

    Hugging Face · Intermediate

    Free courses on how language models, embeddings and agents work, at an engineer's level

Practise

  • Run a small model locally as the heart of the practice app you will defend and test

  • Build your practice app's two tools as a tiny MCP server, so later stages have something real to secure

Keep handy

Also worth your time

Before moving on: Your own small LLM app in Git with a system prompt, retrieval over a few of your documents and two harmless tools, plus a diagram of its context assembly with every way in and every way out marked, and the three legs of the trifecta noted per tool.

Tauseef's advice
Build before you break or defend. People who have written a RAG app and a tool-calling loop themselves understand AI risks in an afternoon; people who have only read about them argue in circles. Keep asking one question: if the model is fooled right now, what is the worst it can do? Your job, on either path, is to make that answer boring.

Mark this stage as not done?