Inspect AI — Framework for Large Language Model Evaluations
by UK AI Security Institute
The eval framework a national AI security institute actually uses, with 200+ prebuilt evals in the box.
Overview
Inspect is built by the UK AI Security Institute together with Meridian Labs, released under MIT, and carries 2.6k stars and 670 forks on GitHub. Its documentation is organized as Basics, Components, Models, Scoring, Agents, Tools, Running, Analysis and Extensions, plus separate reference, evals and changelog areas. The mental model is four primitives that compose: a dataset is a collection of labelled samples with input and target fields; a solver produces an answer for a sample and ranges from a bare model call to a full agent scaffold; a scorer grades that output by text comparison, model grading or custom logic; and a task binds the three into a runnable evaluation. On top of those, the framework provides built-in ReAct and Deep Agent implementations, multi-agent primitives, and tools for bash, Python, text editing, web search, browsing and computer control. Sandboxing is unusually broad — Docker, Kubernetes, Modal, Proxmox and Vagrant — which is what makes it viable for evaluations where the model executes untrusted code. It supports over 20 model providers including OpenAI, Anthropic, Google and Hugging Face, with local inference through Hugging Face, vLLM and SGLang. Developer tooling includes a VS Code extension for authoring and debugging evaluations and the Inspect View web log viewer for monitoring and visualizing runs. More than 200 pre-built evaluations ship ready to run against any supported model, so you can benchmark before writing eval code of your own.
At a Glance
- Topic
- Frameworks
- Level
- Intermediate
- Format
- Documentation
- Cost
- Free
- Duration
- ~3-5 hours to work through Basics and Components, then reference
- Provider
- UK AI Security Institute
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Compose an evaluation from datasets, solvers, scorers and tasks as separate swappable pieces
- ✓Write model-graded scorers for outputs that exact string matching cannot judge
- ✓Build agent evaluations using the built-in ReAct and Deep Agent solvers
- ✓Sandbox untrusted model-generated code with Docker, Kubernetes, Modal or Vagrant
- ✓Give agents bash, Python, editing, web search and computer-control tools safely
- ✓Run the same evaluation against 20+ providers or local vLLM and SGLang servers
- ✓Analyze runs with the Inspect View log viewer and the VS Code extension
Highlights
- •Built and used by a government AI security body, so it is designed for evaluations that have to withstand external scrutiny
- •Over 200 pre-built evaluations are included, so you can benchmark a model before writing a single line of eval code
- •Five sandbox backends including Kubernetes and Modal, which is rare in eval tooling and necessary for code-execution evals
- •MIT-licensed with no hosted-service dependency: 2.6k stars, 670 forks, and everything runs locally
- •The solver and scorer split lets you A/B an agent scaffold without rewriting the evaluation around it
Who It’s For
Best For
- ✓Teams that need reproducible, auditable model evaluations rather than vibe checks
- ✓Safety and security researchers running dangerous-capability or code-execution evals
- ✓Engineers benchmarking competing agent scaffolds against the same task set
- ✓Anyone evaluating local open-weight models through vLLM or SGLang
Prerequisites
- •Working Python, including decorators and typed function signatures
- •Familiarity with LLM APIs and what an evaluation dataset looks like
- •Docker or another supported sandbox for the tool-use portions
FAQ
What is Inspect AI — Framework for Large Language Model Evaluations?
Documentation for Inspect, the open-source LLM evaluation framework built by the UK AI Security Institute with Meridian Labs. It teaches four composable primitives — datasets, solvers, scorers and tasks — then layers on agents, tools, sandboxing and log analysis. It supports over 20 model providers plus local inference, and ships more than 200 pre-built evaluations you can run immediately.
Is Inspect AI — Framework for Large Language Model Evaluations free?
Inspect AI — Framework for Large Language Model Evaluations is free to access.
What level is Inspect AI — Framework for Large Language Model Evaluations for?
Inspect AI — Framework for Large Language Model Evaluations is aimed at a intermediate audience. Recommended background: Working Python, including decorators and typed function signatures, Familiarity with LLM APIs and what an evaluation dataset looks like, Docker or another supported sandbox for the tool-use portions.
How long does Inspect AI — Framework for Large Language Model Evaluations take?
Expect roughly ~3-5 hours to work through Basics and Components, then reference. Most learners work through it at their own pace.
What will I learn from Inspect AI — Framework for Large Language Model Evaluations?
You'll learn: Compose an evaluation from datasets, solvers, scorers and tasks as separate swappable pieces; Write model-graded scorers for outputs that exact string matching cannot judge; Build agent evaluations using the built-in ReAct and Deep Agent solvers; Sandbox untrusted model-generated code with Docker, Kubernetes, Modal or Vagrant; Give agents bash, Python, editing, web search and computer-control tools safely; Run the same evaluation against 20+ providers or local vLLM and SGLang servers; Analyze runs with the Inspect View log viewer and the VS Code extension.
Topics
Sources
This page was written from 2 sources, 1 on domains other than inspect.aisi.org.uk.