PyRIT (Python Risk Identification Tool)

What PyRIT is

PyRIT, the Python Risk Identification Tool for generative AI, is an open source framework from Microsoft. The project describes it as a framework built to help security professionals and engineers proactively identify risks in generative AI systems. The repository now lives at github.com/microsoft/PyRIT; the older Azure/PyRIT repository was archived and points there.

Rather than a single scanner, PyRIT is a toolkit you use from Python code or notebooks. Its main building blocks are:

• Targets: the model or application being assessed.
• Attacks and orchestrators: the workflows that run a test, from single prompts to multi-turn conversations.
• Converters: transformations applied to inputs, used to check whether defences hold up when the same request is phrased differently.
• Scorers: ways of judging whether a response shows the behaviour you are testing for.
• Memory: a record of every exchange, so results can be reviewed and reproduced.

Is it worth your time?

It is aimed at people who already understand AI risks and want to automate red teaming, so it suits the later part of an AI security learning path. If you are new, start with the OWASP Top 10 for LLM Applications and a simpler tool, then come to PyRIT when you want to run structured, repeatable exercises against your own app.

As with any testing tool, use it only on systems you own or are authorised to test. Microsoft's own guidance on planning LLM red teaming is a good companion read: it covers scope, who tests, how results are recorded and how they are reported.

Prerequisites

Solid Python, familiarity with notebooks, access to a model endpoint you control, and a clear idea of which risks you are testing for. A spending limit on any paid API is essential.

How to use it well

• Write down the scope and the harms you are testing before you run anything.
• Begin with one target and one simple workflow from the documentation, then add converters and scorers.
• Review the stored conversations yourself; automated scores are a guide, not a verdict.
• Move confirmed findings into regression tests so fixes stay fixed.

Free or paid?

Free and open source under the MIT licence, installed from PyPI. Documentation is at microsoft.github.io/PyRIT.

Common questions

PyRIT or garak? garak is closer to a scanner you point at a model. PyRIT is a framework for building your own red team workflows. Many practitioners use both.

Is it only for Azure? No. It supports many kinds of targets, including non-Microsoft model APIs and your own endpoints.