Patterns for Building Cybersecurity Evals
by Eugene Yan
How seven published benchmarks test whether LLM agents can find and exploit vulnerabilities, and the eval design patterns they share.
Overview
Patterns for Building Cybersecurity Evals is a June 2026 essay by Eugene Yan, a Member of Technical Staff at Anthropic who previously led ML and AI teams at Amazon, Alibaba and Lazada. It reviews seven published benchmarks that measure whether LLM agents can find and exploit software vulnerabilities, and pulls out the design patterns they share. The opening framework splits every cyber eval into four parts: a sandboxed target (usually Docker containers), difficulty inputs ranging from zero-day (code only) to one-day (a description, patch, crash trace or proof of concept), the tools the agent gets, and a grader. Partial credit comes from subtasks along an attack chain: find the bug, reproduce it with a PoC, gain code execution, then reach an attacker goal. Most graders are deterministic and score outcomes, and transcript audits catch reward hacking. The rest of the piece goes benchmark by benchmark: Cybench (40 professional CTF tasks), CVE-Bench (40 critical web-app CVEs with eight attack goals), CyberGym (1,507 OSS-Fuzz memory-safety instances, where a PoC must crash the pre-patch build and run cleanly on the patched one), ExploitGym (898 instances across userspace programs, V8 and Linux kernel privilege escalation), ExploitBench (41 V8 bugs, five grading tiers, six MCP tools), MHBench (40 emulated multi-host networks) and SCONE-Bench (405 exploited smart contracts). Each entry gives the setup, the grading rule and headline results, and the post cites the arXiv paper behind each benchmark. It contains no code.
At a Glance
- Topic
- Agentic
- Level
- Advanced
- Format
- Guide
- Cost
- Free
- Duration
- ~19 min read
- Provider
- Eugene Yan
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓Break a cyber eval into a sandboxed target, difficulty inputs, agent tools and a grader
- ✓Set task difficulty by choosing zero-day or one-day inputs such as patches and crash traces
- ✓Award partial credit along an attack chain from finding a bug to reaching an attacker goal
- ✓Grade exploits deterministically, for example a PoC that crashes pre-patch code and runs cleanly after the patch
- ✓Audit agent transcripts with model judges to catch reward hacking on exploit tasks
- ✓Compare the setups and results of Cybench, CVE-Bench, CyberGym, ExploitGym, ExploitBench, MHBench and SCONE-Bench
- ✓Control contamination by testing on targets published after a model's knowledge cutoff
Highlights
- •One essay lines up seven cyber benchmarks by setup, grading rule and headline results
- •Written by an Anthropic Member of Technical Staff who publishes widely on LLM evals
- •Reports evaluation cost where known, such as CyberGym's more than $40k in API credits and 1,000 H100 GPU hours
- •Shows the harness can matter more than the model: on MHBench, Sonnet 4 went from 3 of 40 networks to 37 of 40 with the Incalmo framework
- •Cites the arXiv paper for each benchmark and includes a BibTeX block for citing the post
Who It’s For
Best For
- ✓AI safety and security engineers building capability evals for agents
- ✓Teams red-teaming coding agents or judging how far to trust published cyber benchmark scores
- ✓Eval engineers looking for grading patterns that transfer to other agent tasks
Prerequisites
- •Familiarity with LLM agent loops, tool calling and agent evaluation
- •Basic security vocabulary: CTFs, CVEs, proof-of-concept exploits and memory-safety bugs
FAQ
What is Patterns for Building Cybersecurity Evals?
A long-form guide by Eugene Yan of Anthropic to how cybersecurity evaluations for LLM agents are built, drawn from seven published benchmarks. It is for engineers and researchers who design or interpret agent evals for vulnerability discovery and exploitation, and it shows you how to pick a sandbox, difficulty settings, tools and a grader for your own.
Is Patterns for Building Cybersecurity Evals free?
Patterns for Building Cybersecurity Evals is free to access.
What level is Patterns for Building Cybersecurity Evals for?
Patterns for Building Cybersecurity Evals is aimed at a advanced audience. Recommended background: Familiarity with LLM agent loops, tool calling and agent evaluation, Basic security vocabulary: CTFs, CVEs, proof-of-concept exploits and memory-safety bugs.
How long does Patterns for Building Cybersecurity Evals take?
Expect roughly ~19 min read. Most learners work through it at their own pace.
What will I learn from Patterns for Building Cybersecurity Evals?
You'll learn: Break a cyber eval into a sandboxed target, difficulty inputs, agent tools and a grader; Set task difficulty by choosing zero-day or one-day inputs such as patches and crash traces; Award partial credit along an attack chain from finding a bug to reaching an attacker goal; Grade exploits deterministically, for example a PoC that crashes pre-patch code and runs cleanly after the patch; Audit agent transcripts with model judges to catch reward hacking on exploit tasks; Compare the setups and results of Cybench, CVE-Bench, CyberGym, ExploitGym, ExploitBench, MHBench and SCONE-Bench; Control contamination by testing on targets published after a model's knowledge cutoff.
Topics
Sources
This page was written from 3 sources, 2 on domains other than eugeneyan.com.