Automating eval design and hillclimbing with Claude
by Anthropic
How to design an eval you can trust and improve against it without overfitting, using claude-api build-eval and hillclimb.
Overview
Automating eval design and hillclimbing with Claude is a playbook on claude.dev by Lance Martin of Anthropic, published September 28, 2026 and about 4,000 words long. The first half is eval methodology that applies with any tooling: tasks should mirror production, scores should rise with model capability and effort, the best model should still have headroom well below 100%, and run-to-run variance should be low. It warns against picking only the cases today's model fails, because that measures one model's failure fingerprint. The second half documents two commands that ship in the claude-api skill (github.com/anthropics/skills): build-eval, which sources cases from production transcripts, tickets and hand-written examples, picks a programmatic or LLM-as-judge grader with a rubric of checkable claims, and runs a baseline with a confidence interval plus grader, plumbing and headroom diagnostics; and hillclimb, which splits train and test, checks eval noise, then proposes one root-cause patch per round and keeps it only if both splits improve. Two worked examples show the method: an internal 44-ticket support benchmark taken from 78.6% to 90.5% on held-out tickets at about one-fifth the cost, and the skill's own docs-derived eval raised from 66.1% to about 88%. Overfitting gets its own section, with mitigations such as never pasting failing transcripts into the prompt.
At a Glance
- Topic
- Agentic
- Level
- Intermediate
- Format
- Guide
- Cost
- Free
- Duration
- ~12 min read, plus hands-on time running the commands in Claude Code
- Provider
- Anthropic
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Apply four tests of a good eval: production-like tasks, capability scaling, headroom and low variance
- ✓Source eval cases from production transcripts, bug reports and a few hand-written examples
- ✓Choose between programmatic checks and an LLM judge that scores a rubric of checkable claims
- ✓Check a grader for consistency by grading the same output twice and reading scored transcripts
- ✓Pick a hillclimbing surface that is cheap to iterate and whose score changes are attributable
- ✓Split cases into train and test sets and revert patches that only improve the train score
- ✓Target cost reduction at equal performance using prompt audits, caching, model choice and effort
- ✓Run /claude-api build-eval and /claude-api hillclimb in Claude Code against your own app
Highlights
- •Written by Lance Martin, with two worked examples that report baseline, held-out score and cost per ticket
- •The methodology half stands on its own for teams that do not use Claude Code
- •Treats overfitting as a first-class problem and lists the specific ways eval fixes leak into a harness
- •The commands ship in the public anthropics/skills repo, so the process can be reproduced on your own eval
Who It’s For
Best For
- ✓AI engineers building their first serious eval set for an LLM app or agent
- ✓Teams tuning prompts, skills or model and effort choices to cut cost without losing accuracy
- ✓Claude Code users who want an eval-and-iterate loop driven from the terminal
Prerequisites
- •Experience shipping at least one LLM application, prompt or agent skill
- •Basic statistics such as confidence intervals and train/test splits; Claude Code for the hands-on part
FAQ
What is Automating eval design and hillclimbing with Claude?
This Anthropic playbook by Lance Martin is for engineers who ship LLM apps, agents or skills and need evals that hold up. It sets out what makes an eval trustworthy, how to validate a grader, and how to hillclimb prompts, skills, model and effort settings against a train/test split, using two Claude Code commands from the claude-api skill.
Is Automating eval design and hillclimbing with Claude free?
Automating eval design and hillclimbing with Claude is free to access.
What level is Automating eval design and hillclimbing with Claude for?
Automating eval design and hillclimbing with Claude is aimed at a intermediate audience. Recommended background: Experience shipping at least one LLM application, prompt or agent skill, Basic statistics such as confidence intervals and train/test splits; Claude Code for the hands-on part.
How long does Automating eval design and hillclimbing with Claude take?
Expect roughly ~12 min read, plus hands-on time running the commands in Claude Code. Most learners work through it at their own pace.
What will I learn from Automating eval design and hillclimbing with Claude?
You'll learn: Apply four tests of a good eval: production-like tasks, capability scaling, headroom and low variance; Source eval cases from production transcripts, bug reports and a few hand-written examples; Choose between programmatic checks and an LLM judge that scores a rubric of checkable claims; Check a grader for consistency by grading the same output twice and reading scored transcripts; Pick a hillclimbing surface that is cheap to iterate and whose score changes are attributable; Split cases into train and test sets and revert patches that only improve the train score; Target cost reduction at equal performance using prompt audits, caching, model choice and effort; Run /claude-api build-eval and /claude-api hillclimb in Claude Code against your own app.
Topics
Sources
This page was written from 3 sources, 2 on domains other than claude.dev.