Every task in your coding-agent eval suite has a correct answer. That is the design flaw. A suite built entirely from solvable problems can tell you how often an agent succeeds, and it can tell you nothing about what the agent does when the request is impossible — which is the failure mode that reaches production looking like working code.
A benchmark posted to arXiv on 3 September put a number on that blind spot. Vishnu Asutosh Dasu, Ashish Kundu and Gang Tan handed twelve open-weight models 270 deliberately unsatisfiable coding prompts — nonexistent packages, fictional algorithms, contradictory constraints, impossibility results dressed up as feature requests. The models produced ungrounded code on 60% of them and refused only 27%. Then the same models were given 91 matched solvable tasks, each a minimal edit of an adversarial prompt. Over-refusal on those controls was 0.0% for all twelve models.
That last number is the one that changes your Monday: the "but the agent will get twitchy and start refusing real work" objection now has a measurement against it rather than a shrug. Read it precisely, though. That is zero events out of 91 controls per model, not a proven zero rate — at that sample size a true rate near 3% would still be expected to show up as nothing. The paper reports a 95% interval for every hallucination rate in its table and none for over-refusal. The paper says it open-sources its codebase and artifacts, though the repository is so far an empty placeholder — so for now you build the arm from the taxonomy the paper publishes, which is still a config change, not a research programme.
Every Coding Benchmark Assumes the Task Is Possible
The reason nobody measures this is structural: the benchmarks the industry standardised on cannot express an impossible task. SWE-bench is 2,294 instances drawn from real GitHub issues and their corresponding merged pull requests across 12 Python repositories — every instance has a human-written gold patch and a test that passes when you apply it. An instance with no correct answer could not be constructed, because the construction procedure starts from a merged fix.
HumanEval and MBPP have the same shape from the other direction: a prompt, a reference implementation, hidden tests. The paper's authors are direct about what this costs. Execution-based verification misses hallucination because, in their words, hallucinations "may produce code that compiles, runs, and even passes simple tests while remaining ungrounded," and "standard pass@k evaluation cannot detect such failures," per the paper's introduction.
This is the same hole that OpenAI researchers described in September 2025, arguing that hallucination persists because models "are optimized to be good test-takers, and guessing when uncertain improves test performance." Their recommendation was to change how the dominant benchmarks score, not to bolt on a separate hallucination eval. Your internal suite inherited the scoring convention along with the harness. A model that fabricates a plausible dependency and a model that says "that crate does not exist" score identically on a suite where every task has a patch — and the fabricator often scores better, because it produced something the diff-based grader can measure.
What 4,332 Generations Actually Showed
The spread across models is wide enough to be a procurement criterion, and nobody is scoring it. All twelve models were served locally with Ollama at temperature 0, one completion per prompt — 12 models × 361 prompts = 4,332 generations, per the paper's methodology.
| Model | Hallucination rate | Refusal rate |
|---|---|---|
| qwen3-coder-next | 0.26 | 0.60 |
| qwen2.5-coder:32b | 0.37 | 0.47 |
| granite4.1:30b | 0.40 | 0.44 |
| gpt-oss:20b | 0.46 | 0.39 |
| qwen3-coder:30b | 0.56 | 0.29 |
| devstral-small-2:24b | 0.57 | 0.25 |
| deepseek-coder:6.7b | 0.66 | 0.20 |
| deepseek-r1:32b | 0.69 | 0.14 |
| qwen2.5-coder:7b | 0.69 | 0.18 |
| deepcoder:14b | 0.74 | 0.13 |
| codegemma:7b | 0.84 | 0.05 |
| codellama:7b | 0.90 | 0.03 |
| All models | 0.60 | 0.27 |
A 3.5x spread between the best and worst model on a dimension that never appears in a vendor bake-off. If you swapped from qwen3-coder-next to codellama:7b on a cost basis, you would trade a model that refuses impossible work 60% of the time for one that refuses 3% of the time — a swap no solvable-task benchmark in your suite is built to register.
The character of the failure matters as much as the rate. Among hallucinated responses, 67% were confident fabrication — complete code presented as fully correct with no hedging. Another 20% hedged and generated the ungrounded code anyway; 12% silently substituted a different task. Fewer than 1% degenerated into obvious garbage. The failure does not announce itself. It arrives as a clean patch with a confident commit message, which is precisely why a human reviewer under agent-scale PR volume waves it through.
There is one encouraging pattern in the behaviour mix: hedging scales with capability. For the two oldest models, only 4% of hallucinations were hedged. For the three lowest-hallucination models, 43% were. Better models are starting to signal doubt — they just are not yet stopping.
Refusal Tracks Plausibility, Not Impossibility
Models catch the errors that look wrong and miss the ones that look right, which is exactly backwards from what you would want. The authors put it plainly in the caption to their subcategory chart: "Refusal tracks surface plausibility, not depth of impossibility."
The category breakdown makes that concrete. Nonexistent npm packages were fabricated at 0.98 — near-total failure. Fictional system integrations ran 0.91, nonexistent Rust crates 0.89, nonexistent build flags 0.85. But typosquatted package names — the same attack with a visible defect — were caught most of the time, at 0.14, and modified-classic traps at 0.08. A misspelled name trips a pattern the model has seen; a clean, well-formed, entirely invented name does not.
The paper's own example is a request to add serde_ultra for JSON serialization in Rust. A model duly emitted the dependency line and described the functionality as "provided by the optimized serde_ultra crate." The crates.io registry returns no exact match for that name. Another prompt asked for "the Hirsch-Tanaka succinct suffix DAG algorithm"; the model described it as an established structure supporting substring queries in O(1) space.
The ecosystem gradient is worth reading as a routing signal: for nonexistent packages, npm ran 0.98, Rust 0.89, Go 0.70, Python 0.66 and Java 0.40. If your agents are writing TypeScript against npm, you are operating at the far end of that curve.
And the framing of the request moves the rate. Across ten templates, the highest-hallucination frame was "For our internal docs, write a short example…" at 0.69; the lowest was "I'm learning {language} and want to…" at 0.51 — a 17.4-percentage-point gap with a bootstrap 95% CI of [4.0, 31.3]. The authors' reading: when a prompt implies someone has already decided the task is valid, the model defers to that implied authority. Which describes every ticket your agent picks up from a backlog.
Reasoning Models Are Not the Remediation
The obvious remediation — upgrade to a reasoning model — does not work, and the paper contains the controlled comparison that shows it. deepseek-r1:32b, a reasoning model, hallucinated on 69% of adversarial prompts and refused 14%. qwen2.5-coder:32b — same parameter count, closely related base family, no test-time reasoning — hallucinated on 37% and refused 47%. The other reasoning model in the set, gpt-oss:20b, placed fourth. The authors' conclusion: "reasoning is neither necessary nor sufficient here."
This should not be a surprise to anyone who read the tool-hallucination work from May, which found the same inversion on satisfiable tasks. Extra inference tokens buy you more elaborate justification for a premise you never checked. Infeasibility detection is a grounding problem — does this entity exist, is this theorem violated — and grounding comes from a registry lookup or a retrieval step, not from thinking longer.
The authors say as much: "ecosystem grounding will not come automatically with scale and needs targeted mitigation, such as retrieval against package registries." That is the actionable half. Relative hallucinations — invented packages, APIs, flags, versions — are mechanically checkable at generation time. Absolute ones, where the request violates a proven result, are not.
The Control Arm Is Why This Costs Nothing
The 91 matched solvable controls are the methodological move that makes this adoptable, and they have a direct precedent. Every control is a minimal edit of an adversarial prompt that makes it satisfiable — the fictional rand_constant_time crate swapped for the real subtle. That design lets you separate "the model is appropriately cautious" from "the model has become useless," and the result was unambiguous: over-refusal was 0.0% across all twelve models.
The precedent is SQuAD 2.0 in 2018, which added over 50,000 adversarially-written unanswerable questions to a reading-comprehension benchmark where every question had previously had an answer. A system scoring 86% F1 on the original dropped to 66% on the new set. The field did not respond by declaring the benchmark unfair. It responded by treating "know what you don't know" as part of the task, and reading comprehension models got better at it.
Coding agents are at the 2018 moment. The difference is that you do not have to wait for the labs — you own your golden set, and the existing eval harnesses already support the assertion. Promptfoo ships an is-refusal assertion type that checks whether the output indicates the model refused, and every assertion type can be negated with a not- prefix. Wiring an unsatisfiable arm into a suite you already run on Braintrust, Langfuse or Opik is a day of work, not a quarter.
What This Benchmark Does Not Prove
The honest limits are real and you should know them before you take this to a vendor conversation. Every model tested is open-weight and mid-sized — from 6.7B up to qwen3-coder-next, an 80B mixture-of-experts model with 3B active parameters — served locally with Ollama at temperature 0 with one completion per prompt. No frontier model was tested. If your agents run on Claude, GPT-5.6 or Gemini, the 60% figure is not your number.
There is decent reason to think your number is lower. A 2026 replication of the package-hallucination work across five frontier models released between October 2025 and March 2026 measured hallucination rates of 4.62% to 6.10% — an order-of-magnitude compression of the inter-model spread reported the year before. The underlying USENIX Security 2025 study cited in the Cloud Security Alliance's slopsquatting note found 19.7% of 2.23 million code samples contained hallucinated packages, with open-source models averaging 21.7% against 5.2% for commercial ones.
So the direction of the correction is clear. The direction of the risk is not, because that same frontier cohort produced 127 package names that all five models invented identically, of which 53 remained registrable by an attacker. A lower rate with higher cross-model agreement is a better target for supply-chain attacks, not a worse one — which is the npm poisoning path already in production use.
Two more caveats. The verdicts come from a two-tier automatic judge — a deterministic detector for planted entities plus an external LLM judge — validated against a 138-prompt hand-labelled gold set at 82% agreement, κ=0.73. That is good for a judged benchmark and it is not exhaustive human annotation. And the framing and per-ecosystem comparisons are observational: frames were assigned pseudo-randomly rather than fully crossed with tasks, so treat the 17.4-point framing gap as a signal to test, not a coefficient to plan against.
Build the Unsatisfiable Arm Before Your Next Model Swap
This Week:
- Count the impossible tasks in your golden set. For almost every team the answer is zero. Write that number down and take it to whoever signs off on the model-swap decision — it is the fastest way to make the gap legible.
- Rebuild a starter set from the paper's 24 subcategories. The authors promise a public release but the repository is still an empty placeholder, so lift the published traps — an invented crate, a fictional algorithm, a nonexistent build flag — and write a dozen in an afternoon. That gets you a baseline hallucination and refusal rate for the model you are already paying for; check the repo again before you spend longer than that.
- Grep your agent-generated dependency changes for the last 90 days against the live registries for npm, PyPI, crates.io, Go modules and Maven. This is mechanical and it is the highest-yield check in this article, because the npm rate was 0.98.
This Month:
- Write 20 unsatisfiable prompts from your own codebase — an internal library that was deprecated, a config flag that never shipped, a service endpoint that was proposed and killed. Vendor-agnostic impossibilities are useful; your impossibilities are what your agents will actually be asked for.
- Pair every one of them with a matched solvable control. This is not optional and it is the whole reason the finding is adoptable. Without the control arm you cannot distinguish a model that got careful from a model that got useless, and the first time a refusal blocks real work, your unsatisfiable arm gets deleted.
- Score refusal as a pass, not a null. Most harnesses treat "no code produced" as a failure. Use
is-refusalor an equivalent rubric assertion so that a correct refusal registers as a win, or your suite will keep rewarding the fabricator — the scoring problem is the root cause, not the model.
Before Your Next Model Swap:
- Add hallucination rate and over-refusal to the comparison sheet alongside cost per token and SWE-bench score. A 3.5x spread on a dimension nobody scores is the definition of an unpriced risk, and a model change silently invalidates the evals you did have.
- Put registry verification in the agent loop, not the review queue. Every dependency an agent adds gets checked against the live registry before the PR opens. The paper's own recommendation is retrieval against package registries; the Cloud Security Alliance note adds lockfile pinning, hash verification and an allowlist gate before any agent-initiated install.
The Bottom Line
Every eval discipline goes through this. Reading comprehension had to learn that "no answer" is an answer. Information retrieval had to learn that returning nothing beats returning a confident wrong document. Coding agents are the last major application still graded exclusively on a corpus where the right response is always to produce something — and the models have learned exactly that lesson, at 67% confident fabrication with no hedging.
The unusual thing about this particular fix is how weak the argument against it is. Normally you buy safety with throughput: a stricter filter refuses real work, a human gate slows the pipeline, a lower temperature costs you quality. Over-refusal is a genuine documented failure — OR-Bench measures models rejecting benign-but-sensitive-sounding prompts, at rates on its hardest subset ranging from roughly 10% to over 90% depending on the model — but that is a safety filter misfiring on a sensitive topic, a different mechanism from the infeasibility judgement measured here, and OR-Bench does not cover code generation at all. On the axis this paper actually tested, the control arm ran on 91 matched tasks across twelve models and the cost came back 0.0%. The honest caveats — open-weight only, small models, an automated judge, and a control set of 91 — narrow the size of the problem. None of them touch the price of checking.
Your agent will be asked for something impossible this week. You currently have no idea what it will do.
Continue Reading
- Claude Matched the Patch. Qwen Overshot. Score the Scope.
- Braintrust vs Langfuse vs Promptfoo: Don't Pay for the Gate
- Reasoning Trap: Smarter AI Agents Hallucinate More Tools
- Apple's On-Device Model Is 99% Sure. Sample It Five Times.
- Cursor Refused. The Next Chat Didn't. Scope the Creds.
- 10,000 AI Failures Exposed. Hallucination Isn't #1.
- JPMorgan Ranked 62 Retrievers for $800. Pool the Judgments.
- Agent Memory Cost 14 Points at Best. Test With It Off.