Inspect (UK AI Security Institute)

What Inspect is

Inspect is a Python framework for evaluating large language models. It was created by the UK AI Security Institute, the UK government body that tests advanced AI systems, and is developed with Meridian Labs. Its documentation describes it as a framework for frontier AI evaluations, covering coding, agentic tasks, reasoning, knowledge, behaviour and multimodal understanding.

An Inspect evaluation is built from a few parts:

• Datasets of samples, each with an input and usually a target.
• Solvers and agents that decide how the model is prompted, including multi-step agents such as a ReAct loop.
• Tools the model can call: bash, Python, text editing, web search, browsing, your own tools and MCP tools.
• Scorers that judge each answer, from exact match to model-graded scoring.

Tool calls can run inside sandboxes such as Docker or Kubernetes, which matters when you test agents that execute code. The Inspect View log viewer and a VS Code extension help you read every step of a run. A separate collection, Inspect Evals, offers over 200 ready-to-run evaluations.

Is it worth your time?

For the red-teaming and evaluation side of AI security, yes. Red-teaming finds new failures; evals measure how often known failures happen and whether a change made things better or worse. Inspect is built for the second job at a high standard, and its sandboxing makes it well suited to agent safety tests.

For a first security eval on a small app, promptfoo is quicker to start with. Move to Inspect when you need custom multi-step agent tasks, careful scoring and logs you can share with others.

How to use it well

• Start by running one existing eval from Inspect Evals against a local model and reading the log.
• Then write a tiny eval of your own: ten samples that check one risk in your own app, with a clear scorer.
• Run agent tasks only inside the provided sandboxes, never on a machine with real credentials.
• Test only models and systems you own or are authorised to test, and watch API spending.

Free or paid?

Free and open source under the MIT licence.