Prompt Injection Red-Team Harness for Tool-Using AI Agents

The problem: agents obey whatever they read

AI agents now read emails, browse web pages, open files and call tools on other people's servers. Every one of those is a way in for prompt injection: instructions hidden in content that the model follows as if they came from its owner. An agent that can read a user's inbox and send email can be told, by a single crafted message, to forward private data to an attacker. Security researchers have shown this against real products repeatedly, and the OWASP Top 10 for Agentic Applications lists agent goal hijack as its first risk.

Large companies run red teams. Most small teams and student projects building agents have no practical way to test their own agent against injections, and no numbers to tell them whether a defence helps or just breaks the product.

What already exists and the gap

• AgentDojo, from ETH Zurich researchers and presented at NeurIPS 2024, is a benchmark environment with realistic tasks and injection test cases across banking, messaging, workspace and travel scenarios. It is excellent research infrastructure, but built around its own fixed environments.
• promptfoo has a red teaming mode that generates attacks against LLM applications from a config file.
• Garak from NVIDIA and PyRIT from Microsoft are open source tools for probing models and AI systems for vulnerabilities.

The gap is a harness aimed at the agent a small team actually built: its own tools, its own MCP servers and its own workflows, measuring security and usefulness together, runnable in CI.

Inspiration

AgentDojo's key idea is worth copying: always measure utility and security together, because a defence that blocks every attack by refusing to do anything is useless. Simon Willison's "lethal trifecta", private data plus untrusted content plus a way to communicate externally, gives you a clean way to classify which of your test agents should be vulnerable and why.

What you build

• Agent developers: point the harness at their agent through a small adapter, choose scenarios and get a report.
• Fake environments: an inbox, a folder of files, a set of web pages and a small shop database, each seeded with clean and poisoned content.
• Recording tools: every tool call is logged, so an attack counts as successful only if a forbidden action actually happened, such as an email sent to the attacker's address.
• Reports: attack success rate and task success rate per scenario, per channel and per defence, across repeated trials.

The technically interesting part

• Deciding success in code. Model judges are not enough here. Define each attack's goal as a checkable event in the fake environment, such as a transfer to a specific account.
• Repeated trials. Agents are not deterministic, so run each case several times and report consistency, not a single pass.
• Defences as experiments. Implement least-privilege tools, approval before outbound actions and a split design where the step that reads untrusted content has no dangerous tools. Measure what each costs in utility.
• MCP as an attack surface. A malicious server can hide instructions in tool descriptions or results. Build a test server that does this.
• Safety of the harness itself. Everything runs against fakes in containers, never against real inboxes or accounts.

How to evaluate it

• Attack success rate and task success rate for every agent and defence, with confidence from repeated trials.
• Coverage: attacks per channel and per lethal-trifecta combination.
• Run time and API cost per full evaluation, and for the fast CI subset.
• A user study where two teams run the harness on their own agents and report what they found and fixed.

Suggested plan

• Weeks 1 to 3: read AgentDojo and the OWASP agentic list, build the fake environments and recording tools.
• Weeks 4 to 7: build three target agents and the adapter, write the first 30 attacks.
• Weeks 8 to 11: scoring, repeated trials, reports and the CLI.
• Weeks 12 to 16: implement the three defences and run the experiments.
• Weeks 17 to 22: MCP attack pack, CI mode, user study.
• Weeks 23 to 26: final report and a public write-up.

Path to a product

Security testing for AI agents is a growing market, and companies shipping agents increasingly need evidence that they have tested for injection. A focused open source harness with a clear report format can grow into a CI service, a consultancy offering or a strong entry point into AI security roles. The write-up of your defence experiments is the kind of work security teams notice.

Scope warning

Never test against systems you do not own. Do not build new jailbreak techniques for public models as the core; focus on indirect injection against agents you control. Keep the environments fake and contained.

Learn what you need on Sefism

• AI agents and MCP roadmap: tool calling, MCP servers, evals and the guardrails stage this project builds on.
• AI agents cheat sheet: prompt injection, the lethal trifecta, least privilege and pass rates on one page.
• OWASP Top 10 for Agentic Applications: the risks your attack library should cover.
• OWASP Top 10 for LLM Applications: model-level risks, including prompt injection.
• promptfoo: compare your harness with an existing red teaming tool.
• MCP Python SDK: for building the malicious test server.

Get help with this idea

The hardest part of this project is designing experiments that are fair to each defence and scoring attacks in code rather than by opinion. Tauseef can help you scope the environments, pick the agents to test and turn the results into a strong report. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session