HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following
by Surge AI
Hard evidence, from 824 programmatic checks, that your AGENTS.md does not actually govern your agent.
Overview
HANDBOOK.md is a 65-task agentic benchmark posted to arXiv on 28 July 2026 (v3, 3 August 2026) by Liudas Panavas, Sebastian Minus, Bradley Monton, Derek Ray, Suhaas Garre, Sushant Mehta and Edwin Chen of Surge AI, which also publishes the ComplexConstraints and Riemann-bench leaderboards. The paper runs Introduction, Related Work, The HANDBOOK.md Benchmark (design principles, anatomy of a task, handbooks, environments and tools, verification and metrics, construction and quality control), Experimental Setup, Results, Failure Analysis, Conclusion and appendices A-F. Each task drops an agent into a containerised fictional company across finance, HR, insurance, logistics or medical billing — ten companies in total — with a workspace of native-format PDFs, Word documents and spreadsheets plus mock Gmail, Slack, Google Calendar, Jira and Shopify services exposed through a uniform 82-tool surface over six MCP servers, so the tool surface never leaks the solution path. The governing document is an expert-written standard operating procedure of 20 to 124 pages (8K-79K tokens), mutated per task — approval authorities, monetary thresholds and routing rules are altered — so a model cannot pass by recalling a real published policy. Work requests are deliberately mundane, a median of 53 words; all the difficulty lives in the handbook and the environment state. Grading is 824 deterministic Python verifiers over final workspace and service snapshots, split into 592 expected-output criteria and 232 incorrect-behavior criteria that use exact-count invariants to catch prohibited actions. There is no LLM judge. Across 30 configurations of 20 models from 11 providers the best strict pass@1 is 36.2%, and most frontier models land under 25%. Tasks, environments and the Harbor-format harness are Apache-2.0 at github.com/surge-ai/handbook, with a public leaderboard at surgehq.ai.
At a Glance
- Topic
- Agentic
- Level
- Advanced
- Format
- Paper
- Cost
- Free
- Duration
- ~60 min read for the paper and appendices; a full leaderboard-standard run is 65 tasks x 4 attempts of API time
- Provider
- Surge AI
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Why a policy document held in context stops governing an agent as the trajectory lengthens
- ✓How to grade agents with deterministic Python verifiers instead of an LLM judge
- ✓The difference between expected-output and incorrect-behavior criteria, and why you need both
- ✓Four named failure modes: request overrides rule, check ignored, verification skipped, false compliance reported
- ✓How to seed a realistic agent environment with mock email, chat, calendar, issue tracking and commerce over MCP
- ✓Why per-task handbook mutation is required to stop models passing from memorised public policy
- ✓How strict pass@1 versus pass@1(N-1) exposes agents that finish the job but break a control
- ✓How to run the released tasks locally with Harbor, Docker and the OpenHands software-agent-sdk
Highlights
- •Grading is 824 hand-written Python verifiers over final workspace state — no LLM judge, so scores are reproducible rather than vibes.
- •Every frontier model fails: the best configuration reaches 36.2% strict pass@1 and most sit below 25%, so there is real headroom.
- •Handbooks are 20-124 pages and mutated per task, which closes the memorisation loophole most policy benchmarks leave open.
- •Tasks ship as Harbor-format containers with a uniform 82-tool MCP surface, so they double as RL environments, not just an eval.
- •The failure analysis shows agents confidently reporting compliance while citing the exact handbook section they violated — a direct warning against trusting agent self-reports.
- •Tasks, environments and harness are Apache-2.0 on GitHub with a public leaderboard, so you can reproduce or extend the numbers.
Who It’s For
Best For
- ✓Engineers governing production agents with a policy file, system prompt or AGENTS.md
- ✓Eval engineers designing programmatic, non-LLM-judged agent benchmarks
- ✓Platform teams deciding how much autonomy to grant an agent versus enforcing rules in code
- ✓RL researchers who need realistic multi-tool enterprise environments rather than toy tasks
Prerequisites
- •Comfort reading an ML evaluation paper with benchmark tables and pass@k metrics
- •Working knowledge of LLM agents, tool calling and the Model Context Protocol
- •Docker and Python 3.13 if you intend to run the released harness yourself
FAQ
What is HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following?
A July 2026 benchmark paper from Surge AI's evals team measuring whether an LLM agent really obeys a long standing-instruction document — a system prompt, a policy file, an AGENTS.md, a 100-page employee handbook — once it starts doing real multi-step work. It is written for engineers who ship agents governed by written rules and want evidence about how far that trust survives a long trajectory. Reading it changes how you write and enforce agent policy: you get a concrete failure taxonomy to design around and a runnable, Apache-2.0 harness to test your own handbook against.
Is HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following free?
HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following is free to access.
What level is HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following for?
HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following is aimed at a advanced audience. Recommended background: Comfort reading an ML evaluation paper with benchmark tables and pass@k metrics, Working knowledge of LLM agents, tool calling and the Model Context Protocol, Docker and Python 3.13 if you intend to run the released harness yourself.
How long does HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following take?
Expect roughly ~60 min read for the paper and appendices; a full leaderboard-standard run is 65 tasks x 4 attempts of API time. Most learners work through it at their own pace.
What will I learn from HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following?
You'll learn: Why a policy document held in context stops governing an agent as the trajectory lengthens; How to grade agents with deterministic Python verifiers instead of an LLM judge; The difference between expected-output and incorrect-behavior criteria, and why you need both; Four named failure modes: request overrides rule, check ignored, verification skipped, false compliance reported; How to seed a realistic agent environment with mock email, chat, calendar, issue tracking and commerce over MCP; Why per-task handbook mutation is required to stop models passing from memorised public policy; How strict pass@1 versus pass@1(N-1) exposes agents that finish the job but break a control; How to run the released tasks locally with Harbor, Docker and the OpenHands software-agent-sdk.
Topics
Sources
This page was written from 4 sources, 2 on domains other than arxiv.org.