garak

What garak does

garak is an open source command-line tool, maintained by NVIDIA, that checks whether a language model can be made to fail in ways you do not want. The project describes it as an LLM vulnerability scanner. It looks for problems such as hallucination, data leakage, prompt injection, misinformation, toxic output and jailbreaks.

It is built from four kinds of plugin:

• Probes generate the test inputs for a particular kind of weakness.
• Detectors decide whether each answer shows the unwanted behaviour.
• Generators connect garak to the model under test.
• Harnesses run the whole process and collect the results.

It can talk to many model sources, including Hugging Face models, OpenAI-compatible APIs, AWS Bedrock, LiteLLM, NVIDIA NIM, llama.cpp style local models and custom REST endpoints.

Is it worth your time?

Yes, once you understand the OWASP Top 10 for LLM Applications at a conceptual level. Running garak against a small model on your own machine shows you, very concretely, what automated AI testing looks like: many probes, a detector verdict for each, and a report you have to interpret. Reading the probe and detector descriptions is itself a good way to learn the risk categories.

Use it only against models and applications you own or are explicitly authorised to test. Pointing a scanner at someone else's AI product without permission is unauthorised testing, whatever the tool.

Prerequisites

Comfort with Python and the command line, and a model to test. A small open model served locally, for example through Ollama, is the cheapest and safest target. If you use a paid API, set a spending limit first, because a full scan sends a lot of requests.

How to use it well

• Start with a small subset of probes rather than everything, so runs are quick and results are readable.
• Read what each failing probe was checking before drawing conclusions.
• Treat a clean result as "no known issues found by these probes", never as proof of safety.
• Turn anything important into a test case in your own eval suite so it is checked on every change.

Free or paid?

Free and open source under the Apache 2.0 licence. It installs with pip.

Common questions

Does it test my whole app or just the model? Mainly the model or an endpoint that wraps it. Application-level risks such as excessive tool permissions still need design review and your own tests.

Can I contribute? Yes. The project accepts contributions such as new probes, detectors and documentation fixes, which makes it a good first open source contribution for an AI security portfolio.