Harness Engineering for Self-Improvement
by Lilian Weng (Lil'Log)
A research map of how agent harnesses are designed, then optimized by agents themselves, from ACE and Meta-Harness to AlphaEvolve and the Darwin Godel Machine.
Overview
Lilian Weng's July 2026 Lil'Log post treats the harness, the code and conventions wrapped around a model that handle planning, tool use, context, artifacts and evaluation, as a lever for recursive self-improvement alongside raw model capability. It has three parts. Harness Design Patterns covers workflow automation as a plan, execute, test, improve loop (with Karpathy's autoresearch repo and the Codex agent loop as examples), the file system as persistent memory for logs, diffs and traces, and sub-agents with background jobs that record status in files so they can recover after interruption. A case study groups the tool set that Claude Code, Codex, OpenCode and Cursor-style agents have converged on: file system, shell, git, MCP and Skills, web search, artifacts, background processes and delegation. Harness Optimization then walks the targets in order of ambition: context engineering (ACE, Meta Context Engineering, Meta-Harness), workflow design (AI Scientist, ScientistOne, Autodata, ADAS, AFlow), self-improving harnesses (STOP, Self-Harness, Agentic Harness Engineering, and Lin et al. 2026 on harness updating versus harness benefit), evolutionary search (Promptbreeder, GEPA, AlphaEvolve, ThetaEvolve, ShinkaEvolve, Darwin Godel Machine, Hyperagents) and joint harness plus weight updates (SIA, Continual Harness). Future Challenges lists seven bottlenecks, including weak evaluators, memory lifecycle, diversity collapse and reward hacking. An appendix summarizes PaperBench, CORE-Bench, ScienceAgentBench, RE-Bench, MLE-bench and KernelBench, and the post ends with 39 references and a BibTeX entry. Weng also wrote the 2023 Lil'Log post LLM Powered Autonomous Agents, and this post reads as its 2026 follow-up on how agents improve themselves.
At a Glance
- Topic
- Agentic
- Level
- Advanced
- Format
- Guide
- Cost
- Free
- Duration
- ~31 min read, plus the 39 linked references if you follow them
- Provider
- Lilian Weng (Lil'Log)
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓The three harness design patterns: workflow loops, file-system memory, and sub-agents with background jobs
- ✓Which tool categories modern coding agents like Claude Code, Codex and OpenCode have converged on
- ✓How ACE, Meta Context Engineering and Meta-Harness optimize context and harness code automatically
- ✓How workflow-search methods such as ADAS and AFlow search over agent designs and graphs
- ✓What self-improving harness methods like STOP, Self-Harness and AHE do to limit reward hacking
- ✓When evolutionary search (GEPA, AlphaEvolve, ShinkaEvolve, Darwin Godel Machine) fits an optimization problem
- ✓How SIA and Continual Harness decide between updating the harness and updating model weights
- ✓The seven open bottlenecks for automated research agents, from fuzzy evaluators to diversity collapse
- ✓Which benchmarks (PaperBench, RE-Bench, MLE-bench, KernelBench) measure research-engineering agents
Highlights
- •Reached 334 points and 80 comments on Hacker News in August 2026
- •Connects day-to-day coding-agent tooling with the 2025-2026 literature on automated prompt, workflow and harness optimization in one reading path
- •Each method gets a short mechanical description, so you can tell ADAS from AFlow from Meta-Harness without opening every paper
- •Ends with a benchmark appendix and 39 references, which makes it usable as a reading list for a research or evals team
Who It’s For
Best For
- ✓Engineers building coding or research agents who want to automate harness and prompt tuning
- ✓Researchers starting work on self-improving agents, automated ML research or agent evaluation
- ✓Tech leads deciding how much agent behavior to encode in harness code versus model training
Prerequisites
- •Working familiarity with LLM agents, tool calling and at least one coding agent such as Claude Code or Codex
- •Comfort reading ML research summaries; the post assumes you know what reward hacking, distillation and evolutionary search are
FAQ
What is Harness Engineering for Self-Improvement?
Harness Engineering for Self-Improvement is a long-form research survey by Lilian Weng, published July 4, 2026, on the system around a base model: planning, tools, context, memory files, sub-agents and evaluation. It is written for AI engineers and researchers who build coding or research agents and want to know which published methods let an agent improve its own prompts, workflows and harness code.
Is Harness Engineering for Self-Improvement free?
Harness Engineering for Self-Improvement is free to access.
What level is Harness Engineering for Self-Improvement for?
Harness Engineering for Self-Improvement is aimed at a advanced audience. Recommended background: Working familiarity with LLM agents, tool calling and at least one coding agent such as Claude Code or Codex, Comfort reading ML research summaries; the post assumes you know what reward hacking, distillation and evolutionary search are.
How long does Harness Engineering for Self-Improvement take?
Expect roughly ~31 min read, plus the 39 linked references if you follow them. Most learners work through it at their own pace.
What will I learn from Harness Engineering for Self-Improvement?
You'll learn: The three harness design patterns: workflow loops, file-system memory, and sub-agents with background jobs; Which tool categories modern coding agents like Claude Code, Codex and OpenCode have converged on; How ACE, Meta Context Engineering and Meta-Harness optimize context and harness code automatically; How workflow-search methods such as ADAS and AFlow search over agent designs and graphs; What self-improving harness methods like STOP, Self-Harness and AHE do to limit reward hacking; When evolutionary search (GEPA, AlphaEvolve, ShinkaEvolve, Darwin Godel Machine) fits an optimization problem; How SIA and Continual Harness decide between updating the harness and updating model weights; The seven open bottlenecks for automated research agents, from fuzzy evaluators to diversity collapse; Which benchmarks (PaperBench, RE-Bench, MLE-bench, KernelBench) measure research-engineering agents.
Topics
Sources
This page was written from 3 sources, 1 on domains other than lilianweng.github.io.