Meta-Harness: End-to-End Optimization of Model Harnesses
by Stanford IRIS Lab, KRAFTON and MIT (Yoonho Lee, Omar Khattab, Chelsea Finn et al.)
Let a coding agent rewrite your agent's harness: search over memory, retrieval and scaffold code using full execution traces as feedback.
Overview
Meta-Harness: End-to-End Optimization of Model Harnesses (arXiv 2603.28052, submitted March 30, 2026) is by Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab and Chelsea Finn of Stanford, KRAFTON and MIT. Khattab created DSPy, so the paper comes from people who have built prompt optimizers before. Its argument is that a system's performance depends heavily on the harness around a frozen model, and that text optimizers such as OPRO, TextGrad and GEPA compress feedback too much to improve harness code well. Meta-Harness runs an outer loop in which a proposer (Claude Code with Opus 4.6 and a small domain skill) reads a filesystem holding the source code, scores and execution traces of every earlier candidate, using grep and cat, then writes a new harness that is evaluated and logged. The paper estimates about 10M tokens of diagnostic context per iteration. Section 4 tests three domains. In online text classification it reaches 48.6% average accuracy against ACE's 40.9% with less context. In retrieval-augmented math, one evolved retrieval harness raises accuracy on 200 IMO-level problems by 4.7 points on average across five models it was not searched on. On TerminalBench-2 its harness scores 76.4% with Claude Opus 4.6 and 37.6% with Haiku 4.5. Appendices show the discovered harnesses, proposer file-access statistics and practical implementation tips. Code is MIT-licensed on GitHub with an onboarding flow for new domains.
At a Glance
- Topic
- Agentic
- Level
- Advanced
- Format
- Paper
- Cost
- Free
- Duration
- ~2 hours to read the paper and appendices; longer to run the reference examples in the repo
- Provider
- Stanford IRIS Lab, KRAFTON and MIT (Yoonho Lee, Omar Khattab, Chelsea Finn et al.)
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Why the harness around a frozen model (storage, retrieval, prompt construction) can move results as much as the model
- ✓Why score-only text optimizers like OPRO, TextGrad and GEPA lose the feedback needed to fix harness code
- ✓How to set up an outer loop where a coding agent proposes harnesses from prior code, scores and traces
- ✓How to structure a filesystem of candidate harnesses so a proposer can grep failures back to specific design decisions
- ✓How evolved harnesses compared with ACE, few-shot, BM25 and dense retrieval baselines on classification and math
- ✓What a discovered TerminalBench-2 coding harness changes in system prompts, tool definitions and completion checks
- ✓How to watch for overfitting when the search set and the evaluation benchmark overlap, as the authors discuss
Highlights
- •The official stanford-iris-lab/meta-harness repo has about 1.7k stars and ships two reference experiments plus an ONBOARDING.md for applying it to your own domain
- •Reports concrete numbers in three domains: 48.6% vs ACE's 40.9% on online classification, +4.7 points on IMO-level math, 76.4% on TerminalBench-2 with Opus 4.6
- •Appendices include the actual discovered harnesses and implementation tips, so you can study what the search produced
- •It is candid about limits: only one proposer agent was tested, the TerminalBench-2 search and evaluation share tasks, and a higher leaderboard entry could not be reproduced
- •The optimized TerminalBench-2 harness is published as a separate artifact repo you can run
Who It’s For
Best For
- ✓Engineers building agent scaffolds or coding agents who want to optimize the harness automatically
- ✓RAG and retrieval engineers looking for a method to evolve retrieval and context-construction code
- ✓Researchers working on prompt optimization, DSPy-style compilers or self-improving agents
Prerequisites
- •Experience building LLM agents or RAG pipelines, including prompt and context management
- •Familiarity with prompt optimizers such as DSPy or GEPA helps with the related-work comparisons
- •To run the code: Python with uv, Claude Code (or an adapted proposer) and model API access
FAQ
What is Meta-Harness: End-to-End Optimization of Model Harnesses?
Meta-Harness is a March 2026 paper from Stanford, KRAFTON and MIT that automates harness engineering, the code that decides what an LLM stores, retrieves and sees. It is for engineers who build agents, RAG pipelines or coding scaffolds and want to search over that code automatically instead of hand-tuning prompts and context rules, with open-source code to try it.
Is Meta-Harness: End-to-End Optimization of Model Harnesses free?
Meta-Harness: End-to-End Optimization of Model Harnesses is free to access.
What level is Meta-Harness: End-to-End Optimization of Model Harnesses for?
Meta-Harness: End-to-End Optimization of Model Harnesses is aimed at a advanced audience. Recommended background: Experience building LLM agents or RAG pipelines, including prompt and context management, Familiarity with prompt optimizers such as DSPy or GEPA helps with the related-work comparisons, To run the code: Python with uv, Claude Code (or an adapted proposer) and model API access.
How long does Meta-Harness: End-to-End Optimization of Model Harnesses take?
Expect roughly ~2 hours to read the paper and appendices; longer to run the reference examples in the repo. Most learners work through it at their own pace.
What will I learn from Meta-Harness: End-to-End Optimization of Model Harnesses?
You'll learn: Why the harness around a frozen model (storage, retrieval, prompt construction) can move results as much as the model; Why score-only text optimizers like OPRO, TextGrad and GEPA lose the feedback needed to fix harness code; How to set up an outer loop where a coding agent proposes harnesses from prior code, scores and traces; How to structure a filesystem of candidate harnesses so a proposer can grep failures back to specific design decisions; How evolved harnesses compared with ACE, few-shot, BM25 and dense retrieval baselines on classification and math; What a discovered TerminalBench-2 coding harness changes in system prompts, tool definitions and completion checks; How to watch for overfitting when the search set and the evaluation benchmark overlap, as the authors discuss.
Topics
Sources
This page was written from 5 sources, 3 on domains other than arxiv.org.