AI security: secure and test LLM apps and agents

Stage 2 · Lesson 5 of 7

Probabilistic behaviour and what it means for security

How LLM apps and agents work, and where they break

The same prompt can produce different answers on different runs, because models sample from probabilities, and providers update models over time. For security this has three consequences. A defence that works on ten tries may fail on the hundredth, so results must be measured as rates, not single yes or no answers.

A test that passed last month may fail after a model update, so security tests have to run again on every change. And guards built from models share the same property: they are probabilistic too. This is why this roadmap keeps separating deterministic controls, such as code that checks permissions, which hold every time, from probabilistic ones, such as , which reduce risk.

Common trapreporting "I tried it and the defence held" after a handful of attempts.

A question to check yourself, an AI practice session and 1 cheat sheet entry come with the full roadmap. Unlock

Resources for this stage

Start with· follow one of these

  • CoursePaid
    Grokking Prompt Engineering for Software Engineers

    Design Gurus · Beginner · Subscription

    How system prompts, context and structured output really work, the ground every LLM risk stands on

  • CourseFree
    Hugging Face Learn

    Hugging Face · Intermediate

    Free courses on how language models, embeddings and agents work, at an engineer's level

Practise

  • Run a small model locally as the heart of the practice app you will defend and test

  • Build your practice app's two tools as a tiny MCP server, so later stages have something real to secure

Keep handy

Also worth your time

Mark this stage as not done?