Security foundations for AI work

The general security ideas every AI security task stands on. The Cybersecurity roadmap at /roadmaps/cybersecurity/ covers them in more depth.

Also called threat model, Threat Modeling Manifesto, STRIDE

What: A structured way to find risks early by answering four questions: what are we working on, what can go wrong, what are we going to do about it, and did we do a good job?
Why: It finds design flaws before code exists, and gives every later AI control a reason and an owner.
Example: A diagram of user, app, model, vector store and two tools, with STRIDE and the OWASP lists as prompts for "what can go wrong".

Watch out

Writing a long document once and never updating it when a tool or data source is added.

Also called untrusted input, boundary

What: A line where data or control passes between parts that deserve different trust, such as the internet and your server. Everything crossing from the less trusted side must be checked.
Why: Most bugs are a missed boundary; in AI apps the model sits on one, so its output is untrusted input to the next component.

Watch out

Drawing the model inside the trusted zone because your own company runs it.

Also called principle of least privilege, need to know

What: Each person, service, agent and tool gets only the access its job needs, for only as long as it needs it.
Why: A manipulated model acts with its tools' permissions, so narrow permissions cap the damage of any successful attack.

Watch out

One powerful shared service account for every feature because it was quicker to set up.

Also called authn, authz, access control

What: Authentication checks who someone is; authorisation checks what they may do. Both must be enforced in code on every request.
Why: In AI apps, every tool call and retrieval must be authorised for the real signed-in user, not decided by the model.

Watch out

Checking identity at login and never again when the agent acts later.

Also called API keys, tokens, credentials

What: Keeping keys, passwords and tokens in a secrets manager or server-side environment, narrowly scoped, rotated and out of code, prompts and logs.
Why: AI projects leak secrets through Git, notebooks, system prompts, tool descriptions and agent sandboxes.

Watch out

Putting an API key in a system prompt because users cannot see the prompt.

Also called layered security, secure defaults

What: Stacking independent controls so one failure does not become a breach: identity, scoped data, narrow tools, encoding, guards, limits, logs and a kill switch.
Why: Some AI controls, such as model-based guards, fail a measurable share of the time, so layers are essential.

Watch out

Adding one impressive guardrail product and leaving the other layers empty.

Also called injection, XSS, broken access control

What: Injection, cross-site scripting and broken access control, now often arriving through model output instead of a form field.
Why: Several OWASP LLM risks are these problems in a new place, so proven fixes such as encoding and parameterised queries still apply.
Example: A model answer rendered as raw HTML is XSS; a generated SQL query run as is, is injection.

Watch out

Learning AI security without ever learning the web basics.

Also called SSRF, cloud metadata

What: Tricking a server into fetching an address the attacker chooses, such as an internal service or a cloud metadata endpoint.
Why: Agents that fetch URLs, and MCP clients following OAuth discovery links, are natural SSRF targets.

Watch out

A fetch tool with no allow list and no block on private IP ranges.

Also called scope, authorisation to test, RoE

What: Written permission and scope for a test: which systems, methods, harm types, rates, cost limits, data handling, times and escalation contacts.
Why: Testing without permission is unauthorised access in most countries, public chatbots included.

Watch out

Starting an automated scan before agreeing a spend limit with the owner.

Go deeper