Wiping the Transcript Was the Worst Retry. Paraphrase.

Your agent harness pastes the failed tool call back into the transcript so the model can learn from it. On small models, the model reads it as a template to copy.

A paper posted to arXiv on 24 August, Feedback That Backfires by Esmail Gumaan, measures that directly for the first time. Over a fixed set of four candidate actions — the failed call, the correct one, and two other plausible wrong ones — the normalised probability of re-emitting the exact call that just failed rises from 0.06 to 0.54 once the failure record is added to context. Chance on that set is 0.25. Every one of the six instruction-tuned checkpoints tested showed the effect. And the two remedies most teams reach for first — a "do not repeat that call" line in the system prompt, and wiping the transcript to retry from a clean context — measured as statistically null and as the worst option tested, in that order.

This is not a model problem you wait out with a better checkpoint. It is a formatting decision in code you own.

What "Corrective Gain" Actually Measures

Corrective gain is the change in the log-probability that a model re-emits the exact action that just failed, once the failure record is in its context. The paper defines it as G(a×) = log π(a×|C) − log π(a×|C ⊕ (a×,o×)) — the difference between the model's odds of writing that call before it saw the failure and after. Positive means the record discouraged repetition. Negative means the record made repetition more likely, which the author calls feedback inversion.

Every checkpoint came back negative on the ToolShed environment:

Model	Parameters	Corrective gain (nats)
SmolLM2-135M	0.14B	−23.92
SmolLM2-360M	0.36B	−18.62
Qwen2.5-0.5B	0.49B	−13.91
Qwen3-0.6B	0.60B	−21.50
Llama-3.2-1B	1.24B	−12.41
SmolLM2-1.7B	1.71B	−13.91

These are not exotic research artifacts. SmolLM2 ships in 135M, 360M and 1.7B sizes explicitly because they are "lightweight enough to run on-device", and Qwen2.5-0.5B advertises structured JSON output as a headline capability — which is precisely why it ends up behind the cheap tier of a routed agent stack. If you route classification, extraction or simple tool dispatch to a sub-2B model to hold your inference bill down, this is your checkpoint class.

The Error Text Barely Matters. The Call Does.

The damage comes from the shape of the text, not the meaning of the error. The paper decomposes the inversion into a surface-form term — the failed call sitting in context as a token sequence the model can copy — and a semantic term, the effect of marking that call as failed. The surface-form term accounts for 83% of the damage. The semantic term is small and its sign is not even stable: slightly positive on tool calls, slightly negative on program repair. Median ratio favouring surface form over semantics is roughly 6:1 on tool calls and 61:1 on program repair.

Read that decomposition carefully, because it is what makes the paper actionable rather than merely interesting. If the error message were doing the damage, you would rewrite your error messages — a long, uncertain project across every tool you own. It is not. The damage is the presence of a copyable string. That is a rendering choice in one function.

Your Framework Ships This as the Default

The behaviour the paper indicts is a default in a widely deployed agent framework. LangGraph's tool node formats an invocation failure with a template that interpolates the arguments straight back into the message the model reads: "Error invoking tool '{tool_name}' with kwargs {tool_kwargs} with error:\n {error}\n Please fix the error and try again.". That is the template on the default path: LangGraph's stock error handler catches argument-validation failures and hands back exactly that message, arguments included. Opt into broader catching with handle_tool_errors=True and you get the generic "Error: {error}\n Please fix your mistakes." instead. The default is the one that echoes the failed arguments verbatim. Both then append the natural-language correction instruction the paper measures as inert.

The Claude API has the same shape for a different reason. Tool errors are returned as a tool_result block with "is_error": true, and "tool result blocks must immediately follow their corresponding tool use blocks in the message history" — so the assistant's original tool_use block, complete with the input object that failed, is structurally required to sit adjacent to the error. The same page notes that on an invalid tool request the model "will retry 2-3 times with corrections before apologizing to the user." That is the same structure — retry against a transcript that still holds the failed call — but it is a bounded retry that terminates, on a model far outside the range the paper tested. Do not read it as the same mechanism.

Nobody designed this to be harmful. It is the obvious way to build a ReAct-style agent harness: append what happened, append what the runtime said, ask for the next step. The paper's contribution is showing that "append what happened" has a cost nobody had priced.

---

Why Wiping the Context Made It Worse

The standard fix for a contaminated transcript — delete the failed step and retry clean — was the worst harness in the study. In rollouts on Qwen2.5-0.5B, the "drop" harness looped in 67% of runs against 29% for the standard verbatim transcript, drove the exact repeat rate to 80% (a paired difference of +49 points, interval [+33, +64]), and moved task success by −8 points [−21, +0]. It cost more and bought nothing.

The author's explanation is the sort of thing that is obvious only in retrospect: "Deleting the failed step restores the exact context that produced the failure. A deterministic policy in an identical context emits an identical action, so clean restart does not remove the problem, it guarantees it."

That is worth sitting with if your incident runbook says "clear context and retry." Under greedy decoding — which is what you use when you want reproducible agent behaviour — a clean restart is not a fresh attempt. It is the same attempt. You have deleted the diagnosis and kept the hazard. This is the same trap as agent memory that costs accuracy while feeling like an upgrade: the intervention is intuitive, the measurement disagrees, and nobody checks because the failure looks like the model being dumb.

It also explains something a retry cap cannot fix. Capping the retry budget limits how much a loop costs you, and that is a real and necessary control. It does not stop the loop from converging on the same bad call — it just stops the bill sooner. Same for iteration budgets on repair loops. A budget bounds the damage; it is not a correction.

What Worked, and What It Did Not Buy

The fix that worked is a rendering change with negative token cost. Replace the verbatim failed call with a runtime-generated description built from the error metadata — the paper's example renders [attempt 1 failed: create_event, an argument was badly formatted; that call is not repeatable] — and 76% of the inversion disappears, greedy exact reproduction drops to 0%, and loop rate falls from 29% to 8%. The description is shorter than the call plus its error, so context length goes down.

Harness	Loop rate	Task success
verbatim (standard)	29%	0.42
+ "do not repeat" instruction	17%	0.58
abstract (paraphrased failure)	8%	0.33
abstract + decoder ban	4%	0.33
drop (clean-context retry)	67%	0.33

A decoder-level ban on previously-failed strings stacks on top, cutting the exact repeat rate from 31% to 7% at, in the author's words, no cost in generated tokens. This one is available today if you self-host: vLLM's bad_words sampling parameter blocks "the last token of a corresponding token sequence... when the next generated token can complete the sequence", and logit_bias is there too. If you are already choosing between vLLM, TensorRT-LLM and SGLang, this is a capability worth checking before you commit.

Now the part a vendor would leave out. Neither surface-form fix improved task success. The paper says so plainly — the abstraction moved success by −8 points [−21, +0]. On the model tested, 18% of actions were prose where a tool call was expected, which no arrangement of the transcript repairs. What you buy is the elimination of wasted loops, not accuracy. The author is blunt about the distinction: "Removing the loop returns the agent's budget to it; whether it can spend that budget well is a question about the model, not the harness."

The instruction result deserves a steel-man. At the probe level, adding "do not repeat a call that has already failed" to the system prompt changed the log-probability of repeating by +0.07 nats, interval [−0.07, +0.21] — indistinguishable from nothing, consistent with the long-standing finding that telling a model not to produce something keeps that something in context. But in rollouts, that condition posted the best task success of any harness, 0.58 against 0.42 — a paired difference of +17 [+4, +33] points, an interval clear of zero. The author reports it and explicitly declines to build a mechanism on it: "with 24 tasks the effect is real but the explanation is not identified, and the most plausible readings, that the instruction makes the model more careful in general or less willing to abandon a task early, are not about repetition at all." Treat it as a real but unexplained effect on task completion, not as evidence about looping — and do not let anyone tell you the instruction was measured as harmful. It was measured as irrelevant to repetition.

Where This Finding Stops

Do not extend this to your frontier model. The tested range is 135M to 1.7B parameters. A log-linear fit across those six points has a slope of 9.51 nats per decade of parameters (R² = 0.64) and crosses zero at 37.6B — and the author reports that number in order to argue against it. Refitting with each model held out moves the crossing point between 20B and 79B, a factor of 3.8 decided by which six checkpoints happened to be affordable. The honest statement is that two things could be true at 8B or 70B and this data cannot separate them: the copying term might keep shrinking, or the semantic term might start to dominate.

The nearest evidence above the tested range points the other way, and the article would be dishonest without it. An April 2026 study of iterative self-repair across seven models from Llama-3.1-8B to Llama-3.3-70B found execution feedback universally improved pass rates — by 16 to 30 points on MBPP, which is one of the two environments this paper uses. Both can hold: showing a model its error helps once the model is large enough to read the error, and hurts while it is still small enough to only copy it. But it means the 1.7B-to-8B span is an unmeasured gap between two opposing results, not a trend line you can draw through.

The variance inside the tested range matters too. Byte-for-byte greedy reproduction of the failed call ran at 43% on SmolLM2-135M and 41% on SmolLM2-360M, but at 3% on Qwen2.5-0.5B and 0% on both Qwen3-0.6B and Llama-3.2-1B — while the largest checkpoint tested, SmolLM2-1.7B, ran at 27%. The log-probability inversion was universal; the visible looping was not, and it does not fall away cleanly with size. Two checkpoints of adjacent size behaved completely differently, which is the strongest argument for measuring your own rather than trusting the table.

And the rollouts are small: 24 tasks, one model, six steps each, with 95% cluster bootstrap intervals that touch zero on several comparisons. The probe results across six checkpoints are the solid part. The rollout table is directional.

What to Do Before Your Next Agent Run

This Week:

1. Find the function in your harness that renders a tool failure into the next turn's context. Read what it emits. If the failed arguments appear verbatim, you have the defect described here — no measurement needed to confirm that much.
2. Check whether any retry path in your stack deletes the failed step and re-runs. If it does, that is now the highest-priority item on this list, because it is the one measured as actively worse than doing nothing.
3. Pull the exact-repeat rate out of your existing agent traces. Count the runs where consecutive tool calls were byte-identical. You almost certainly have the data and are not looking at it.

This Month:

1. Change the rendering: emit a runtime-generated description of the failure instead of the call. You have the tool name, the error class and the argument that broke — that is enough to write the sentence. It is a smaller diff than any prompt-engineering project you are currently running.
2. If you self-host, wire the previously-failed call string into bad_words or a logit bias for the next decode. It costs no tokens and it is a belt over the braces.
3. Add exact-repeat rate as a first-class metric next to success rate in whatever you use for agent monitoring. Loop rate and success rate move independently — the paper's own table shows a harness that cut loops by 3.6x and lowered success. If you track only one, you will make the wrong call.

Before Your Next Model Swap:

1. Re-run the probe on the new checkpoint. The measurement is teacher-forced scoring over a few hundred probe items with key/value cache reuse, which removes 87% of the token positions a cache-free implementation would score, and ran on a single laptop CPU. This is a half-day of engineering time, not a research programme — the same posture that applies to auditing a small on-device model you did not train.
2. Do not assume the previous checkpoint's behaviour carries over. SmolLM2-360M and Qwen2.5-0.5B sit 0.13B apart and differed by 38 percentage points on exact reproduction.

The Bottom Line

Every generation of automation rediscovers that the log is not the fix. We spent a decade learning that stack traces piped back into an automated remediation system produce confident, repeated, wrong actions, and we built runbooks and circuit breakers in response. Agent harnesses threw that away and went back to "show it the error and ask again," because the model can read English and it felt like it should work. On small models it does not, and the reason is almost banal: a language model asked to continue a document will continue it, and the failed call is the most recent thing in the document.

The correction is not more instruction. It is not showing the model the thing you do not want it to write. Describe the failure. Do not reprint it.

Continue Reading

• Copilot Retried Into GitHub's Outage. Cap Your Agents.
• Your Terraform Repair Loop Broke Passing Checks. Stop at 3.
• Agent Memory Cost 14 Points at Best. Test With It Off.
• What Is an Agent Harness? The $1B Claude Code Architecture
• vLLM vs TensorRT-LLM vs SGLang: Default to vLLM
• Apple's On-Device Model Is 99% Sure. Sample It Five Times.
• Reasoning Trap: Smarter AI Agents Hallucinate More Tools
• Agents Averaged 73. Only 30% Were Usable. Grade Pass/Fail.