The loop your platform team wired up to fix infrastructure misconfigurations is also quietly un-fixing them. In a new study of 4,440 LLM repair iterations on Terraform, 3.3% of scenarios hit at least one security control that had been passing and then failed — a control nobody asked the model to touch, broken while it fixed something else. (Some of those recover later in the run; more on that below.)
The more useful finding is the second one. The repair loop's value peaks at iteration three. Iteration four buys 0.3 percentage points. Iteration five gives some of it back. If your remediation tool is configured to iterate until the scanner goes green — and most of them are — you are paying tokens after iteration three to make your infrastructure marginally less secure.
What This Study Actually Measured
It measured the raw per-iteration security trajectory, which nobody had looked at before. Benjamin Agyekum and Fabio Santos ran 5,968 scenario timelines from the IaC-Eval benchmark through up to five repair iterations each, across 15 configurations — six RAG cells analysed per model, nine non-RAG cells (Zero-Shot, Few-Shot, Chain-of-Thought) pooled across both models, at three temperatures each — producing 4,440 iteration transitions with scanner results on both sides.
The reason this had not been measured is a reporting convention. The IaC literature reports cumulative-best metrics — pass@k, best-of-n — which are, as the authors put it, "monotonically non-decreasing by construction." A loop that fixes three checks and breaks one still shows as improvement. The cumulative curve went from 73% to 83% and looked like a clean win. The raw trajectory underneath it does not.
They define a security regression precisely: "Given consecutive iterations i and i+1 of a scenario where both have Checkov results, a regression occurs for check c if c ∈ PASSED(i) and c ∈ FAILED(i+1)." The study used Checkov v3.2.392 as its security oracle against 30 CIS Benchmark check IDs — one open-source Terraform scanner standing in for the whole class.
Two numbers come out, and they differ by roughly 4x. Under standard detection — any check that moves from the passed set to the failed set — 13.8% of scenarios and 24.8% of transitions regress. Under strict detection, which requires the check to be exclusively passing before and exclusively failing after, the rate drops to 3.3% of scenarios and 5.2% of transitions. Roughly 76% of standard-mode flags are multi-resource bookkeeping, not a genuine control failure.
Lead with 3.3%. The authors call it "the conservative, defensible rate," and a headline built on 13.8% would misrepresent their own paper. But note what they recommend operationally: report the standard figure as "the inclusive outer bound an operator should act on." You are the operator. Both numbers are yours.
Iteration 3 Is Where the Curve Turns
The pass rate climbs, flattens, and then declines — and it flattens earlier than any shipping tool assumes.
| Iteration | Pass rate |
|---|---|
| 0 | 73.2% |
| 1 | 74.7% |
| 2 | 81.7% |
| 3 | 83.1% |
| 4 | 83.4% |
| 5 | 82.6% |
Iteration 3 → 4 gains 0.3pp. That rate is the share of individual checks passing, so across ~30 tracked checks it is under a tenth of one check — below the authors' 0.5pp materiality threshold, which they are honest enough to list among their own validity threats as "subjective." Iteration 5 loses 0.8pp against the peak.
Now put the risk curve next to it. Per-transition regression rates run 22.8% for 0→1, 26.6% for 1→2, and peak at 29.5% for 2→3. The transition that delivers most of your remaining value is also the riskiest one in the run. That is not an argument for stopping at two — the pass-rate jump from 74.7% to 83.1% across iterations 1–3 is the whole point of the loop. It is an argument for stopping at three with a diff in hand.
This lines up with independent work. A verifier-first evaluation of agentic IaC generation published in May 2026 found that with a budget of four attempts, "83% of GPT-4o tasks pass in 1–2 attempts; 16% exhaust the full budget." The tail does not converge. It grinds.
The Model Rewrites the Block Instead of Editing the Line
Resource restructuring causes 79.0% of regressions. Configuration drift accounts for 15.5%, outright argument removal 3.6%. The failure mode is not the model deleting an encryption flag on purpose. It is the model regenerating an entire resource block to satisfy one failing check and not carrying every attribute forward.
The evidence for that is in the diff size. Regression transitions change 140.1 lines on average versus 53.8 for clean ones — 2.6x the churn, Cohen's d = 0.90. A large effect, from the simplest possible mechanism: the more the model rewrites, the more it drops.
What breaks depends on which detection mode you read, and this is where the two numbers diverge most. Under standard detection, access control is 38.5% of regressions, data protection 25.9%, networking 14.2%, encryption 13.5%, logging 5.8% — and the three most frequently regressing checks, CKV_AWS_356, CKV_AWS_111 and CKV_AWS_109, together 886 events and 33.6% of the standard-mode total, all govern IAM privilege boundaries. Wildcard resources, missing policy conditions, role trust boundaries. Under strict detection the ranking inverts: networking leads at 42.6%, encryption is 21.6%, and access control drops to 9.2%. So the IAM concentration belongs to the inclusive outer bound, not to the 3.3% headline — but the outer bound is the one the authors tell you to act on, and a loop that widens a permission while fixing an unrelated bucket setting is the same class of problem as the cross-agent escalation path where every call was individually authorized.
One honest caveat on those specific checks: the IAM wildcard rules have a documented false-positive history. A since-closed issue against the scanner records that logs:DescribeLogGroups genuinely requires Resource: "*", and that constraining it to an ARN leaves a policy that "does not grant access to anything." That issue names CKV_AWS_356 and not the other two, so it does not cover the whole 886 — but some fraction of those events may be the model correctly writing a policy the scanner dislikes. The authors flag this themselves: a static scanner is "a proxy for security," not security.
Check Volatility Is the Abort Signal You Already Have
If you instrument one thing, instrument the number of checks that change state per iteration. It is the strongest predictor in the paper and it costs nothing.
Regression transitions show check volatility of 11.55 versus 2.38 in strict mode — a 4.9x difference with Cohen's d = 1.49, the largest effect size the study reports. In standard mode it is 5.17 versus 2.09. The authors' own recommendation: "Check volatility is cheap to compute online and is the strongest regression signal... Halting or flagging a repair loop when volatility spikes offers regression protection at negligible cost."
You already have this data. Every scan emits a pass/fail set. The delta between consecutive sets is a subtraction.
Do not halt on a single regression, though. Of 2,639 standard-mode regressions, 967 — 36.6% — self-correct within an average of 1.2 iterations, and 28.5% of scenarios show checks oscillating in and out of the passed set. A one-strike halt would abort a third of the runs that were about to fix themselves. Halt on a volatility spike; flag a single regression and let the loop run to its budget.
Two configuration findings are worth a note and no more. Chain-of-Thought had the lowest non-RAG regression rates at 8.2–9.6%, versus 11.6–15.5% for zero-shot. RAG configurations produced zero strict-mode scenario regressions — 0 of 1,297 for one model, 0 of 929 for the other. The authors are careful that this ordering reflects model composition rather than a proven causal effect of the prompting strategy, and so am I.
The Case for Iterating Anyway
The strongest version of the other side is that iteration is the only thing that makes LLM-generated infrastructure usable at all, and the evidence for that is overwhelming.
On the same benchmark, GPT-4 achieves a pass@1 accuracy of 19.36% against 86.6% on an equivalent Python benchmark — the 458 human-curated AWS scenarios in IaC-Eval are simply harder than the Python tasks models are trained on. The FSE 2026 DPIaC-Eval work measured six frontier models at 20.8–30.2% first-attempt deployment success, and its IaCGen framework lifts that to "54.6–91.6% generated IaC templates from all evaluated models deployable in the first 10 iterations." Ten iterations. Twenty-five with human feedback.
And a March 2026 self-healing infrastructure paper reports a closed-loop multi-agent system hitting 96.8% drift detection, 95.2% security misconfiguration detection, and a 6.9-minute mean time to remediation — measured on a Terraform dataset of simulated drift situations, not production estates. Vendors market the same shape: one platform describes the agentic IaC cycle as Observe, Detect, Plan, Validate, Apply, Confirm, and notes that "the loop running continuously is what separates this from a script."
The sharpest counter is TerraRepair, a July 2026 agent that repairs Checkov findings in Terraform — the same scanner, the same cloud, the same task — with a budget of ten steps per finding, not three. It lifts scanner-verified fix rates from 26.6% to 78.4%. If ten steps beat one by 51 points, capping at three looks like leaving value on the table.
None of that is wrong. Here is why it does not settle the question.
TerraRepair's ten steps are not this study's five. Its agent queries the dependency graph and the provider schema, re-runs the scanner in-loop, and escalates rather than guessing — which is the emit-a-patch discipline recommended below, not the blind regeneration that drives 79% of these regressions. It also never checks whether a previously-passing control broke; nor does any other paper here. That is the shared blind spot, and it is the whole point.
The rest optimize for deployability and correctness, not for the security controls that were already passing before the loop started. DPIaC-Eval measured the security side of its own output at an 8.4% compliance rate and 25.2% user-requirement coverage. The verifier-first study does gate on policy, via opa eval — but it asks whether a task ends compliant, never whether a control that was already passing stopped. And an August 2026 security-first evaluation of text-to-Terraform found the two properties are close to orthogonal: one open model produced 77.8% syntactically valid Terraform and zero scanner compliance, while the best closed model managed 23.1% policy compliance at the strictest level. A loop that converges on "it deploys" is not converging on "it is safe."
What the Study Does Not Prove
It does not prove your loop regresses at 3.3%. Take the mechanism, not the constant.
Only two models were tested — Gemini 2.0 Flash and Mistral Large Latest — and neither is frontier. The gap between them is enormous: standard-mode regression ran 32.6–41.2% for Mistral Large against 1.9–4.7% for Gemini Flash, an odds ratio of 17.29. The authors note that "more recent... models may restructure less aggressively," which is precisely the variable that drives 79% of the regressions.
Everything else narrows it further. Terraform and one scanner only — the paper concedes that "other IaC tools (Ansible, CloudFormation) and validators (tfsec, Terrascan) may exhibit different patterns." AWS only. Single runs per configuration, so point estimates "may vary ±2–5pp." Root-cause classification came from automated diff analysis that "was not validated against human labels." And 902 scenario-configuration pairs — 13% — are missing outright to API errors and timeouts.
What generalizes is the shape: regeneration causes regression, churn predicts it, volatility predicts it harder, and the value curve flattens before the loop stops. That holds regardless of which model you have wired in. Test the constants on your own corpus. This is the same discipline as re-running your evals when a vendor silently swaps a model — the benchmark is a starting hypothesis, not your number.
What to Change in Your Loop
The whole point of this study is that the fix is configuration, not architecture. Every item below is a setting or a subtraction.
This Week:
- Find the iteration cap. Open your remediation tool's config and locate the max-attempts value. If it is unset, unbounded, or "until validation passes," you do not have a budget — you have a while-loop. Write down what it currently is.
- Diff the full control set, not the failing check. Most loops re-scan and ask "did the target check pass?" Store the complete pass/fail set at every iteration and compare set-to-set. A regression is invisible to any check that only looks at the thing it was asked to fix.
- Log lines-changed per iteration. You are looking for the 140-line rewrite in a queue of 50-line edits. It is the cheapest proxy for restructuring risk and you can add it with a
git diff --stat.
This Month:
- Cap iterations at three and measure what you lost. Run a week of remediations at three and at your current ceiling. If the delta in resolved findings is under half a percent, keep the cap permanently.
- Add a security anchor to the repair prompt. The authors' first recommendation is that models "should be constrained to make minimal modifications during repair rather than regenerating entire resource blocks." Instruct the model to emit a patch, not a resource. Then verify it did — a prompt is not an enforcement mechanism, which is the same lesson as deny rules versus polite instructions in coding agents.
- Wire volatility into the loop as an abort. Compute the count of checks changing state per transition, set a threshold from your own baseline, and halt above it. Do not halt on one regression — a third of them self-correct.
- Make regressions visible in the feedback. The second recommendation is check-aware feedback: tell the model explicitly which previously-passing checks it just broke. Most loops feed back only the remaining failures, so the model has no idea it caused one.
Before Your Next Renewal:
- Ask your remediation vendor for the per-iteration curve, not pass@k. If the only number they will give you is cumulative-best, they are reporting a metric that is non-decreasing by construction and cannot show you this failure. That question alone tells you whether they have looked.
- Get the iteration cap and the abort signal into the product config, not the documentation. A default you cannot change is a default you own the consequences of.
- Require a full-control-set diff in the audit trail for every automated change. If a control silently flipped in production, you need to know which iteration did it, and you will not reconstruct that later.
The Bottom Line
This is the same lesson the industry keeps relearning at a different layer of the stack. Amazon's AI-generated code caused outages and the answer was a human sign-off gate. AI SRE agents closed 35,000 incidents and the open question was always what else they touched. Sixty-four percent of the Fortune 500 use AI coding agents and a third measure the result. With 42% of committed code AI-written or AI-assisted on Sonar's own 2026 developer survey, which projects 65% by 2027, the loop is not going away and should not.
The specific trap here is that the metric everyone reports cannot show the failure. A cumulative-best curve rises whether your loop is fixing things or trading them. It looked like a 73-to-83 improvement, and it was — with 3.3% of runs quietly knocking out a control that had been passing on the way. You would never see it, because nothing in the pipeline was watching the checks that were already green. That is a measurement problem before it is a model problem, and it is exactly the kind of confident-looking output that reviewers wave through.
Cap it at three. Diff the whole board. Watch the volatility.
Convergence is not a safety property. It is just the loop running out of things to say.
Continue Reading
- Amazon's AI Code Created Outages. Now Humans Must Sign Off.
- One Agent Escalated Another. Every Call Was Authorized.
- An AI SRE Agent Fixed 35,000 Incidents While Engineers Slept
- Claude Code Stops Asking Aug 14. Prompts Aren't Policy.
- DeepSeek Swapped the Model. Your Eval Didn't Notice.
- Alibaba's Agent Coded 16 Days. A Human Wrote 13 Commits.
- Best AI Agent Monitoring: Langfuse, Then a Real Kill Switch
- The Vaguer the AI Explanation, the More Novices Trusted It