Give a junior analyst ChatGPT for a specification review and you get fewer defects found, no time back, and an analyst who gets good at the job more slowly.
That is the measured outcome of a controlled crossover experiment submitted to arXiv on 21 August 2026 by Giovanna Broccia, Julian Frattini, Chetan Arora, Maurice ter Beek, Alessandro Fantechi, Andreas Vogelsang and Alessio Ferrari. Thirty-four people inspected real specifications with and without LLM support. The assisted condition found fewer defects, took the same amount of time, and — the part nobody is pricing — left the assisted cohort learning the task more slowly than the cohort that did it unaided first.
If you are rolling an AI review assistant out to your business analysts and junior engineers this quarter, this is the study that tells you which stage of the review it is allowed to touch.
What the Experiment Actually Measured
Requirements inspection is the practice of reading a written specification line by line to find defects before anyone writes code against it. The experiment measured whether an LLM assistant helps a human do that.
Thirty-four bachelor's students in computer engineering at the University of Florence were split into two groups of seventeen and inspected two comparable game specifications — an Arkanoid document with 40 requirements and 21 seeded defects, and a Snake document with 39 requirements and 19. Group 1 inspected without ChatGPT in period one and with it in period two; group 2 did the reverse, with a one-week washout between sessions. Participants recorded their own start and end times.
The defects they hunted are requirements smells: patterns in the text that signal a likely problem. Nine types across three categories — lexical (subjective language, optional parts, weak verbs), syntactic (vague pronouns, passive voice, negative phrases) and semantic (logical inconsistencies, numerical discrepancies, ambiguities). They come out of the requirements quality factor ontology that Frattini and colleagues assembled from 105 primary studies. Each flagged smell also had to be classified as nocuous — genuinely capable of producing divergent interpretations — or innocuous, resolved by surrounding context.
Participants used ChatGPT on GPT-4o and GPT-4.1 family models, with the exact model depending on their own subscription tier. They were taught basic prompting principles and then given free rein. No template, no scripted workflow, no retrieval scaffolding. That is worth holding onto — it is both the study's biggest limitation and the most accurate simulation of what actually happens when you hand a team ChatGPT and a deadline.
The Result That Should Change Your Rollout Plan
Detection accuracy fell. Observations from the unassisted condition achieved about 8% higher F₁ scores on average, and the authors state the finding plainly: "LLM support negatively affects smell detection accuracy but has no significant effect on smell classification or task duration."
Hold that number loosely, though, because the paper does. On detection, the authors report that "the total uncertainty around these averages, represented by the 95%-credibility intervals, still overlaps." Eight percent is the observed gap in this sample, not a tightly bounded effect — at thirty-four participants the interval is wide enough to include no difference at all. The direction is the authors' stated conclusion; the magnitude is not something to put in a business case.
What survives that caveat is the part that matters commercially. The trade most teams assume they are making — a little accuracy for a lot of throughput — was not on offer. Task duration showed no significant difference between conditions; those intervals overlapped too. Classification accuracy, once a smell had been correctly spotted, was unaffected either way.
So the assistant did not make reviewers faster, did not sharpen their judgement about severity, and on the one dimension where it shifted the average at all, it shifted it the wrong way. There is no measurement in this experiment on which it won.
The authors' own reading of why is uncomfortable and familiar: participants "may have delegated part of the inspection effort to the LLM, relying on its responses instead of engaging deeply with the requirements." This is the same mechanism as the finding that vague AI explanations draw more trust from novices than detailed ones — the reviewer stops reviewing and starts ratifying.
The Learning Penalty Is the Expensive Half
The accuracy gap is a quarter's problem. The learning gap is a career's.
Both groups improved between period one and period two — a normal maturation effect. But they did not improve equally. The group that learned inspection without ChatGPT first improved its detection F₁ by roughly 12% between periods. The group that started with ChatGPT improved by roughly 7%. The same asymmetry showed up in classification: about 5% improvement for the unassisted-first group against about 3% for the assisted-first group. These are reported as observed differences between the two sequences, without credibility intervals of their own, and the authors read the same asymmetry as "a clear indication of a carryover effect" — the standing hazard of crossover designs. It is simultaneously the paper's most consequential result and its least tightly bounded one.
The authors' conclusion: "early reliance on LLMs may hinder the development of inspection skills, potentially by short-circuiting the reflective processes through which novice inspectors internalize inspection strategies." And more directly: "Introducing LLMs too early in inspection training may limit students' opportunities to develop foundational inspection skills and critical reading abilities."
This is a cost that will not show up in any dashboard you currently run. Review throughput looks the same. Defect escape rates move slowly and are confounded by a dozen other things. What degrades is the rate at which your junior people become senior people — and you will notice that in about three years, when the bench you assumed you were building is not there. The labour market is already splitting between roles that command an AI premium and entry-level roles that do not, and a training pipeline that produces slower learners is not a neutral operating choice.
Why the Benchmarks Said This Would Work
The steel-man here is genuinely strong, and it is worth stating properly: the model is capable of this task in isolation.
The paper's own related-work section catalogues the evidence. Fantechi et al. showed GPT-3.5 identifying certain classes of inconsistency. Mahbub et al. found GPT-4 better at identifying incomplete requirements than inconsistent or ambiguous ones, while concluding LLMs cannot replace human analysts in zero-shot settings. Bashir et al. reported a 20.2% average improvement in classification performance with ten-shot prompting over zero-shot on industrial datasets. Vendors have built on exactly this: Jama Connect Advisor claims to enhance "the quality and clarity of requirement language per INCOSE and EARS standards, guiding those new to writing requirements" and to reduce "a major source of defects", while QRA's QVscribe offers "language-aware objective quality scoring" and rewrite guidance inside the same tool.
None of that is refuted by this experiment. What the experiment shows is that model capability on a benchmark and workflow outcome with a human in the loop are different quantities, and the second one is the one you are buying.
That gap keeps reproducing. DORA's 2025 report found AI adoption among software professionals at 90%, with over 80% reporting improved productivity and 59% reporting improved code quality — all self-reported. When METR ran a randomised trial instead, 16 experienced open-source developers took 19% longer on 246 issues in their own repositories while believing AI had sped them up by 20%. Microsoft Research and CMU surveyed 319 knowledge workers across 936 real tasks and found higher confidence in the AI predicted less critical thinking, while higher self-confidence predicted more.
Three different populations, three different methods, one pattern: perception says up, measurement says flat or down. DORA's own framing — AI as "a mirror and a multiplier" of what a team already does — fits this study exactly. It multiplied a novice's willingness to defer.
What the Study Does Not Prove
Take the limitations seriously, because a leader who over-reads this will make a different bad decision.
Thirty-four participants is a small sample, and the authors say so. They were bachelor's students without extensive practical inspection experience — the population the finding is about, but not the population of a senior systems engineer at a medical device firm. The specifications were simpler than industrial ones, and each requirement contained at most one seeded smell, where a real specification stacks interacting defects. Participants worked from a predefined list of nine smell types rather than doing fully open-ended detection. About 2.9% of tasks were completed outside supervised class time.
Two more caveats matter commercially. The models were the GPT-4o and GPT-4.1 families as available in May 2025 — a real generation behind current frontier models. And participants used a raw chat interface, not a purpose-built requirements tool with a fixed rubric, a structured diff and a scored output. It is entirely plausible that a constrained product like QVscribe or Jama Advisor produces a different result than free-form chat. Nobody has run that experiment yet, which is precisely the problem: your vendor is asserting it.
METR's own February 2026 update is the model of intellectual honesty here — it disclosed severe selection bias in its follow-up study and now believes developers are likely more sped up in early 2026 than its 2025 estimate suggested. Treat single studies as evidence, not verdicts. But treat vendor self-report as neither.
Where the Assistant Belongs Instead
The authors give you the placement, and it is the most actionable line in the paper: organisations "should therefore consider restricting their use to specific phases (e.g., post-inspection review or justification refinement) rather than as a primary aid during initial defect detection."
Human first, unaided, to a completed defect list. Then the LLM, on a second pass, against that list — what did I miss, what did I flag that is actually innocuous, is my justification for calling this nocuous defensible. The authors extend the same logic to training: LLMs "can still play a valuable pedagogical role if used deliberately, for example, as tools for discussion, reflection, or comparison after an initial manual inspection."
That ordering preserves the thing the experiment says gets destroyed — the reflective pass where a novice actually reads the specification — while keeping the model's recall as a safety net. It is the same structural argument as keeping the writer and the reviewer independent in AI code review: the value of a second opinion collapses when it arrives before the first one is formed.
What to Do About It
This Week:
- Find out which stage your AI review assistant currently sits at. If it runs before or during first-pass human inspection — an inline score while authoring, a pre-populated defect list at review open — you are running the condition that lost accuracy in this experiment. Move it behind the human pass.
- Pull the actual review-cycle-time numbers for the last two quarters. If your business case was "reviews get faster," this study says that is the one benefit you should be least confident in, and you have the data to check it yourself.
This Month:
- Run a small internal replication. Two comparable specifications, two teams, one with the assistant and one without, seeded defects you control, F₁ scored against your own ground truth. You do not need thirty-four people to learn whether it helps your documents. This is the same discipline as testing an agent with its memory turned off or running an actual holdout instead of quoting a lift number.
- Change what your first-year analysts are told. Unaided inspection first, tool second, for at least their first two quarters — and make that an explicit training policy rather than a preference somebody quietly overrides under deadline.
- Pick one metric that is not throughput. Escaped requirements defects per release, or defect-detection rate against seeded controls. Throughput is exactly where this study says the effect is not, and measuring output volume rather than outcome is the standing failure mode of AI productivity reporting.
Before Your Next Renewal:
- Ask the vendor for a human-in-the-loop study, not a model benchmark. The question is not "what F₁ does your model achieve on requirements smells" — it is "what F₁ do reviewers using your product achieve versus reviewers who are not." If they cannot answer, that is the answer, and it should shape the term length you agree to.
- Ask what the product does when the human disagrees with it. A tool that makes deferral the path of least resistance is buying you the exact effect measured here. Weight that alongside the seat-versus-consumption pricing questions you are already asking.
The Bottom Line
Every previous automation wave in software engineering was sold on the same premise: the tool handles the routine part, the human does the judgement. Static analysers did deliver that. Compilers did. The difference with an LLM assistant is that it does not stop at the routine part — it produces a confident, plausible answer to the judgement question too, and a novice cannot tell the difference between an answer and an answer worth checking.
That is why the placement matters more than the procurement. The same model, at the wrong stage of the workflow, converts a reviewer into an approver. And the time your team appears to save has a way of not being there when you look for it.
The assistant is not the problem. The order is.
Continue Reading
- Same Vendor Wrote and Reviewed 208,145 PRs. Split Them.
- The Vaguer the AI Explanation, the More Novices Trusted It
- Agent Memory Cost 14 Points at Best. Test With It Off.
- Agents Averaged 73. Only 30% Were Usable. Grade Pass/Fail.
- Zalando Auto-Approves a Third of PRs. Agents Made Them Bigger.
- Airbnb Shipped 80% More Features. One Number Is Auditable.
- AI Saves 11 Hours a Week. Workers Waste 6.4 Babysitting It.