Prompt Injection Regression Test Harness for RAG Support Chatbots

The problem: every prompt change is a silent security change

Small product teams are shipping support chatbots that answer from their own help centre and customer data. They tune the system prompt, swap to a cheaper model, add new documents or give the bot a tool, and each change can quietly undo an earlier fix. A bot that refused to reveal another customer's order last week may reveal it after a model upgrade. Unlike ordinary code, there is rarely a test that fails when this happens.

Prompt injection is the top risk in the OWASP Top 10 for LLM Applications (2025), and indirect injection through retrieved documents is especially relevant to support bots, which read content that customers or outsiders can influence. Because no single fix removes the risk, teams need to measure it continuously, the same way they run unit tests.

What already exists and the gap

• promptfoo is an open source tool for LLM evals and red teaming that runs in CI; OpenAI announced in March 2026 that it is acquiring the company, and says the open source project continues.
• garak, from NVIDIA, is an open source scanner that probes models for weaknesses such as data leakage and prompt injection.
• PyRIT, from Microsoft, is a framework for automating red team runs against generative AI systems.
• Lakera Guard, now part of Check Point, is a commercial runtime protection layer against prompt injection and data leakage.

These are powerful but general. The gap for an FYP is a focused harness for one very common shape of app, a RAG support bot with per-customer data, with test cases that understand retrieval and tenancy, a clear score per release and a report a small team can read in five minutes.

Inspiration

The OWASP LLM Prompt Injection Prevention Cheat Sheet recommends layered defences and ongoing testing rather than one filter. Microsoft's guide to planning red teaming for LLMs describes the cycle this project automates: build a list of harms, test it in rounds, record every example so it can be reproduced, and separate finding problems from measuring them.

What you build

• A reference bot you own, so every test is legal and repeatable.
• A test suite where each case names its risk, its input or planted document and the safe behaviour expected.
• A runner that executes the suite against a bot endpoint, applies detectors and produces a score per category.
• CI integration that blocks a change which makes the score worse.
• A report viewer for comparing runs.

Write test cases descriptively and keep them about your own bot's data. The aim is to check that defences hold, not to publish attack text.

The technically interesting part

• Indirect injection through retrieval. Plant documents in the reference bot's own knowledge base and check whether the bot's behaviour changes when they are retrieved.
• Tenancy. Seed two fake customers and check that no answer ever contains the other customer's identifiers, however the question is phrased.
• Judging answers. Rule-based detectors are reliable but narrow; a model-graded check is flexible but can be wrong. Measure the grader against a hand-labelled sample and report its error rate.
• Non-determinism. The same input can give different answers, so decide how many repeats make a result trustworthy.

How you will evaluate it

• Scores before and after each of three hardening changes to the reference bot.
• Agreement between the model grader and human labels on at least 100 results.
• Runtime and cost of a full suite run, and how that shapes what runs on every pull request versus nightly.

Path to a product

Every team shipping an LLM feature needs regression tests for AI risk, and most do not have them. A hosted version that connects to a team's staging bot, keeps suites per product and posts results on pull requests is a clear product shape, as is a consultancy offer to write and maintain suites for clients. The harness plus a published write-up is also a strong portfolio piece for AppSec and AI security roles.

Scope warning

Test only the reference bot and systems the team owns. Do not build a general jailbreak collection, do not target public chatbots, and keep runtime protection out of scope: this project measures, it does not filter.

Learn what you need on Sefism

• Cybersecurity roadmap: choose the AI security path for how LLM apps fail, the OWASP LLM risks with their defences, and testing and guardrails.
• Cybersecurity cheat sheet: the AI security edition covers prompt injection, excessive agency, output handling, the AI supply chain and security evals in a line or two each.
• OWASP Top 10 for LLM Applications: the 2025 list your project maps its checks to.
• MITRE ATLAS: techniques, mitigations and case studies to cite in your threat model.
• promptfoo: compare your harness's design with an established eval and red team tool.
• garak: see how probes and detectors are organised in an open source scanner.
• NIST AI Risk Management Framework: frame your scores as the Measure function in your report.

Get help with this idea

The hard part of this project is not the runner; it is writing test cases and judges you can trust. Tauseef can help you choose the risk categories, design the reference bot and plan an evaluation examiners will respect. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session