AI security cheat sheet (Red-teaming and evaluation)

1. 1Security foundations for AI workThe general security ideas every AI security task stands on. The Cybersecurity roadmap at /roadmaps/cybersecurity/ covers them in more depth.10 concepts

   1. 1Threat modelling (four questions)A structured way to find risks early by answering four questions: what are we working on, what can go wrong, what are we going to do about it, and did we do a…
   2. 2Trust boundaryA line where data or control passes between parts that deserve different trust, such as the internet and your server.
   3. 3Least privilegeEach person, service, agent and tool gets only the access its job needs, for only as long as it needs it.
   4. 4Authentication and authorisationAuthentication checks who someone is; authorisation checks what they may do.
   5. 5Secrets managementKeeping keys, passwords and tokens in a secrets manager or server-side environment, narrowly scoped, rotated and out of code, prompts and logs.
   6. 6Defence in depthStacking independent controls so one failure does not become a breach: identity, scoped data, narrow tools, encoding, guards, limits, logs and a kill switch.
   7. 7Classic web weaknesses in AI appsInjection, cross-site scripting and broken access control, now often arriving through model output instead of a form field.
   8. 8Server-side request forgeryTricking a server into fetching an address the attacker chooses, such as an internal service or a cloud metadata endpoint.
   9. 9Rules of engagementWritten permission and scope for a test: which systems, methods, harm types, rates, cost limits, data handling, times and escalation contacts.
   10. 10Severity for AI findingsRating a finding by likelihood and impact, recording the success rate over many attempts and judging impact by what the app can do.

   10 in this chapterOpen this chapter
2. 2How LLM apps and agents breakFollow a request through an LLM app and an agent to see where text gets in and where actions get out.7 concepts

   1. 1Context assemblyThe text an app builds for the model: system prompt, history, retrieved documents, tool descriptions and results, and the user's message.
   2. 2Instructions and data in one channelA model reads instructions and data as the same stream of text and decides by probability which parts to follow.
   3. 3Retrieval permissionsFiltering retrieval by who is asking, in the query itself, and controlling who can write to indexed sources.
   4. 4Agent loopA loop where the model asks for a tool call, code runs it, the result returns to the context, and the model decides again.
   5. 5Non-determinismThe same input can give different outputs across runs and model versions.
   6. 6Lethal trifectaSimon Willison's name for three capabilities that together let an injection steal data: private data, untrusted content and a way to communicate out.
   7. 7Practice target appA small LLM app you own: a local or spend-capped model, a system prompt, retrieval over your own documents and two harmless tools.

   7 in this chapterOpen this chapter
3. Risk maps: OWASP, ATLAS, NIST, SAIFThe shared vocabularies of AI security. The OWASP LLM Top 10 2025 is still the current edition as of October 2026.7 conceptsOpens with credits
4. Prompt injection and the guard layersPrompt injection as an attack class and the layers that reduce it. No single control is enough; design limits the damage.10 conceptsOpens with credits
5. Agent, tool and MCP securityWhat agents can do, and how to keep it narrow. MCP details follow the current stateless specification; check your SDK's revision.11 conceptsOpens with credits
6. Data and model supply chainRisks that arrive before a user types anything: the data, the model files and what leaks from both.10 conceptsOpens with credits
7. Testing, red-teaming and evalsHow AI systems are tested responsibly, and the main open source tools by purpose. Point them only at systems you own or are authorised to test.11 conceptsOpens with credits
8. Monitoring and incident responseSeeing what an AI app does in production, switching off harm quickly and learning from each incident.7 conceptsOpens with credits
9. Career: roles, certifications and portfolioWhere the work is and how to show you can do it. Certification facts are as of October 2026; check official pages before paying.7 conceptsOpens with credits
10. 10Glossary, A to ZEvery concept in the sheet and the smaller terms it uses, in one list.135 terms

Secure LLM apps and agents, test models safely.