Inspect AI — Framework for Large Language Model Evaluations
by UK AI Security Institute (AISI)
Learn to write real LLM and agent evals with the open-source framework the UK AI Security Institute uses.
Overview
Inspect is a framework for frontier AI evaluations developed by the UK AI Security Institute (AISI) with Meridian Labs, covering coding, agentic tasks, reasoning, knowledge, behavior, and multi-modal understanding. Its documentation teaches the framework's composable building blocks — datasets, solvers that generate model responses, scorers that grade outputs, and agents that run multi-turn tool-using loops — then moves into running evaluations at scale, sandboxing untrusted code with Docker or Kubernetes, and analyzing results. Agents can be equipped with bash, python, text editing, web search, web browsing, and computer tools, and the ecosystem ships more than 200 pre-built evaluations (Inspect Evals) you can run against any model provider. Supporting tooling includes Inspect View, a web-based UI for monitoring and visualizing eval runs, and a VS Code extension for authoring and debugging. The project launched in May 2024 and is under active development with a public changelog, making the docs a current, authoritative reference for evaluation engineering.
At a Glance
- Topic
- Frameworks
- Level
- Intermediate
- Format
- Documentation
- Cost
- Free
- Duration
- Self-paced — tutorial in ~1 hour, full reference ongoing
- Provider
- UK AI Security Institute (AISI)
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Structure an evaluation as a task with a dataset, solver, and scorer using Inspect's composable primitives
- ✓Build agentic evals with multi-turn tool use (bash, python, text editing, web search and browsing, computer use)
- ✓Run untrusted model-generated code safely in Docker or Kubernetes sandboxes
- ✓Score outputs with model-graded and programmatic scorers, and analyze runs in Inspect View
- ✓Reuse and adapt the 200+ pre-built evals in the Inspect Evals catalog across model providers
- ✓Extend the framework with custom model providers, tools, and scorers
Highlights
- •Built and maintained by a national AI institute — the same framework used for frontier model evaluations
- •200+ ready-to-run evaluations you can point at any model
- •Inspect View UI and a VS Code extension for authoring, debugging, and visualizing runs
- •Provider-agnostic: works across major model APIs and local models
Who It’s For
Best For
- ✓AI engineers building evaluation suites for LLM and agent products
- ✓Safety and red-team researchers running standardized benchmarks
- ✓ML platform teams adding regression evals to model deployment pipelines
Prerequisites
- •Comfortable Python
- •Basic experience calling LLM APIs
- •Familiarity with prompt engineering and tool calling helps
FAQ
What is Inspect AI — Framework for Large Language Model Evaluations?
Inspect is the open-source LLM evaluation framework built by the UK AI Security Institute and Meridian Labs, and its documentation doubles as a practical course in how to design, run, and analyze evals. It is for AI engineers who need to measure model and agent performance rigorously instead of eyeballing outputs.
Is Inspect AI — Framework for Large Language Model Evaluations free?
Inspect AI — Framework for Large Language Model Evaluations is free to access.
What level is Inspect AI — Framework for Large Language Model Evaluations for?
Inspect AI — Framework for Large Language Model Evaluations is aimed at a intermediate audience. Recommended background: Comfortable Python, Basic experience calling LLM APIs, Familiarity with prompt engineering and tool calling helps.
How long does Inspect AI — Framework for Large Language Model Evaluations take?
Expect roughly Self-paced — tutorial in ~1 hour, full reference ongoing. Most learners work through it at their own pace.
What will I learn from Inspect AI — Framework for Large Language Model Evaluations?
You'll learn: Structure an evaluation as a task with a dataset, solver, and scorer using Inspect's composable primitives; Build agentic evals with multi-turn tool use (bash, python, text editing, web search and browsing, computer use); Run untrusted model-generated code safely in Docker or Kubernetes sandboxes; Score outputs with model-graded and programmatic scorers, and analyze runs in Inspect View; Reuse and adapt the 200+ pre-built evals in the Inspect Evals catalog across model providers; Extend the framework with custom model providers, tools, and scorers.