Terminal-Bench 2.0 and Harbor Documentation
by Terminal-Bench
Score your terminal agent against the benchmark everyone quotes, and learn how the harness decides pass or fail.
Overview
Terminal-Bench evaluates AI agents on realistic command-line work - compiling code, training models, configuring servers - inside isolated Docker containers, and this documentation is the practical guide to running it. Each task is defined by four parts: a natural-language instruction, a containerized environment, a verification test suite that programmatically checks the resulting container state, and an oracle solution demonstrating a valid approach. Because grading inspects state rather than prose, an agent that merely describes a plausible fix still fails. Terminal-Bench 2.0, announced on 7 November 2025 by Mike Merrill and Alex Shaw, is a harder and far more heavily verified rebuild of the original after the community surfaced flawed v1 tasks - a download-youtube task, for instance, that failed on YouTube anti-bot protection rather than on agent capability. Harbor is the official 2.0 harness: install with 'uv tool install harbor', smoke-test with 'harbor run -d terminal-bench/terminal-bench-2 -a oracle -l 5', then swap in a real agent such as claude-code with a '-m' model flag. Cloud sandboxes are supported via '--env daytona' with an '-n' concurrency flag for parallel runs, single tasks via '--include-task-name', and custom agents via '--agent-import-path'. Harbor works with any agent installable in a container and is designed to support SFT and RL improvement loops. The project is Apache-2.0 with roughly 2,500 stars, 904 commits, about 100 contributors and a 1,000-member Discord; note that the 2.0 leaderboard submission process is documented as still being rebuilt.
At a Glance
- Topic
- Agentic
- Level
- Advanced
- Format
- Documentation
- Cost
- Free
- Duration
- ~1-2 hours to a first scored benchmark run; considerably longer to build and tune a custom agent
- Provider
- Terminal-Bench
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Install Harbor and run Terminal-Bench 2.0 against a model inside Docker
- ✓Read a task definition: instruction, container, verification tests and oracle solution
- ✓Register a custom agent through the agent-import-path interface and score it
- ✓Parallelize evaluation across cloud sandboxes such as Daytona using concurrency flags
- ✓Smoke-test your harness with the oracle agent before spending tokens on a real model
- ✓Interpret leaderboard results and compare terminal agents on a common footing
- ✓Design verifiable end-to-end tasks that assert container state instead of grading text
Highlights
- •Tasks are graded by test suites against container state, so a confident-sounding wrong answer still fails
- •Every task ships an oracle solution, giving you a correctness baseline for your own harness
- •Harbor runs any agent that can be installed in a container and supports SFT and RL improvement loops
- •2.0 was rebuilt specifically because the community found broken v1 tasks - the verification story is documented rather than assumed
- •Free and Apache-2.0, with roughly 100 contributors, 904 commits and an active 1,000-member Discord
Who It’s For
Best For
- ✓Engineers building coding, DevOps or shell agents who need an objective score
- ✓Researchers fine-tuning agents with SFT or RL on terminal tasks
- ✓Teams comparing agent harnesses and models before committing to one
Prerequisites
- •Docker installed and running locally, with enough disk for multi-container task images
- •Comfortable on the command line with uv or pip
- •An API key for whichever model you intend to evaluate
FAQ
What is Terminal-Bench 2.0 and Harbor Documentation?
The official docs for Terminal-Bench, the benchmark that evaluates AI agents on real command-line work inside isolated Docker containers, and Harbor, its 2.0 harness. Written for engineers building coding, DevOps or shell agents who need an objective number. You will learn to install Harbor, run the benchmark against a model, read task definitions, and register and score your own custom agent.
Is Terminal-Bench 2.0 and Harbor Documentation free?
Terminal-Bench 2.0 and Harbor Documentation is free to access.
What level is Terminal-Bench 2.0 and Harbor Documentation for?
Terminal-Bench 2.0 and Harbor Documentation is aimed at a advanced audience. Recommended background: Docker installed and running locally, with enough disk for multi-container task images, Comfortable on the command line with uv or pip, An API key for whichever model you intend to evaluate.
How long does Terminal-Bench 2.0 and Harbor Documentation take?
Expect roughly ~1-2 hours to a first scored benchmark run; considerably longer to build and tune a custom agent. Most learners work through it at their own pace.
What will I learn from Terminal-Bench 2.0 and Harbor Documentation?
You'll learn: Install Harbor and run Terminal-Bench 2.0 against a model inside Docker; Read a task definition: instruction, container, verification tests and oracle solution; Register a custom agent through the agent-import-path interface and score it; Parallelize evaluation across cloud sandboxes such as Daytona using concurrency flags; Smoke-test your harness with the oracle agent before spending tokens on a real model; Interpret leaderboard results and compare terminal agents on a common footing; Design verifiable end-to-end tasks that assert container state instead of grading text.
Topics
Sources
This page was written from 3 sources, 1 on domains other than tbench.ai.