The control you bought is the one this experiment just broke. A Nature Medicine paper published on 4 August and written up the same day by MIT put 623 lay people and 153 primary care physicians in front of the same AI image diagnoses, with four different kinds of explanation attached. The physicians barely needed the explanations. The lay people improved a great deal — and they improved by deferring. The explanations the non-experts found most convincing were the vague ones.
If your AI programme was approved on the promise that a human reviews every output, this paper tells you which human, and what you are allowed to put in front of them.
What the Experiment Actually Measured
This was a controlled test of whether an explanation makes a human a better check on a model, run at a scale that supports the answer. The preprint abstract describes two large-scale experiments — 623 lay people and 153 primary care physicians — pairing a fairness-balanced diagnostic model with different explanation styles.
Four conditions, per MIT's write-up: the prediction plus a confidence score and nothing else; a set of visually similar images; a heat map highlighting the regions of the image the model weighted; and a large language model writing out its reasoning in plain language. That last one is the condition every enterprise vendor is selling you. It is the one that behaved worst.
The lay people got more accurate with every explanation style, and the researchers attribute the gain primarily to deference — they were agreeing with the model, not reasoning alongside it. When the model was right, deference looked like competence. When the model was wrong, the same participants were confidently wrong with it, and the participants who deferred most were the ones who performed worst unaided. The physicians held their ground. They were resilient to incorrect explanations, they benefited irrespective of whether the AI was right, and they scored best on the bare prediction with no explanation attached at all.
The Explanation Quality Problem Nobody Priced In
An explanation's persuasiveness runs in the opposite direction from its usefulness, and that is the finding that should change a procurement conversation. MIT reports that non-experts found explanations more convincing precisely when they were vague or generic, and the reader without the training to check took the hedge as expertise.
This inverts how explainability is bought. Vendors in this category sell fidelity: Fiddler AI advertises Shapley values, integrated gradients, counterfactual analysis and per-prediction attribution, and markets that transparency as reduced regulatory risk and increased trust. None of that is wrong as engineering. But the study measures the human on the other end of it, and the human on the other end is not scoring the explanation on fidelity. They are scoring it on whether it sounds authoritative.
An explanation is a persuasion surface before it is an audit artefact. If you cannot tell a good one from a bad one, it can only make you more compliant.
"Those with the least medical knowledge are most likely to be led astray when explainable AI models give an erroneous output," said Stanford's Roxana Daneshjou, quoted in coverage of the paper. Swap "medical" for whatever domain your model operates in and the sentence still holds.
Your Review Seat Is Staffed by the Wrong People
The enterprise version of this failure is structural, not accidental: the whole economic case for AI-plus-review is that the reviewer can be cheaper than the person the model replaced. That is the configuration the study says fails.
Claims adjudication, underwriting triage, tier-1 support escalation, first-pass contract review, fraud queues — in every one of these the deliberate design is a generalist reviewing at volume, because a specialist reviewing every output erases the savings. We have watched firms build exactly this: Group 1 Automotive pooled 700 roles into a virtual F&I desk where cheaper generalists handle more volume with software in the loop, and TD Bank compressed a 15-hour mortgage pre-adjudication review into three minutes with a human sign-off at the end.
The economics of AI assistance make this worse, not better, because assistance helps the weakest reviewer most. In the BCG field experiment run with Harvard researchers, 758 consultants using GPT-4 on tasks inside the model's capability saw quality scores rise 43% for below-average performers against 17% for those above average. Then the same study handed them a task outside the frontier, and the consultants with AI access were 19 percentage points less likely to reach the correct answer than those without it.
The study does not break that 19-point penalty out by skill level, so reading the two numbers together is inference — but it is the same inference the Nature Medicine data supports directly, where the participants who deferred most were the ones who performed worst unaided. The reviewer who gains the most from the model is the reviewer least able to catch it when the task drifts outside what the model handles. And in production, nobody tells you when the task has drifted.
We have already published the version of this that shows up in code: 92% of teams trust AI code scanning while 70% still ship vulnerabilities to production. Trust and verification came apart there for the same reason.
Explainability Has Failed This Test Before
This is a confirmation, not a discovery, and treating it as news is itself part of the problem. In 2021 a team led by Gagan Bansal ran mixed-method user studies across three datasets, pairing participants with an AI of roughly human-level accuracy, and reported the result plainly: while they saw gains from AI augmentation, those gains "were not increased by explanations," and instead "explanations increased the chance that humans will accept the AI's recommendation, regardless of its correctness."
Five years and one generation of models later, the finding survives contact with LLM-written rationales — and gets sharper, because a fluent paragraph is more persuasive than a saliency map. The new contribution is the split by expertise. The old result said explanations increase acceptance. The new one says who that harms.
If you have an explainability line item in an AI budget that was justified as a safety control, it has now failed the same test twice.
The Regulation Already Names This Failure Mode
European law anticipated this specific mechanism, and most compliance programmes are answering the wrong half of it. Article 14(4)(b) of the EU AI Act requires that the people assigned to oversee a high-risk system "remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias)". Article 26(2) puts the obligation on you as the deployer: "Deployers shall assign human oversight to natural persons who have the necessary competence, training and authority, as well as the necessary support."
Competence, training and authority. Not a checkbox, not a queue. The Nature Medicine result is direct evidence about what "necessary competence" has to mean — competence in the domain, not competence in operating the tool. The physicians in the study were protected by clinical training, not by AI literacy.
Expertise buys resistance, not immunity, and there is a study worth putting against this one. When Radiology gave 27 mammography readers deliberately wrong AI suggestions, accuracy fell from 82.3% to 45.5% among the very experienced readers — and from 79.7% to 19.8% among the inexperienced. The gradient matches the Nature Medicine finding. The floor does not. An expert in the review seat makes the control better, not sound.
That distinction cuts against the dominant enterprise response to skills gaps. Deloitte surveyed 3,235 leaders across 24 countries for its 2026 State of AI in the Enterprise report and found insufficient worker skills to be the biggest barrier to AI integration, with 53% naming broad AI-fluency education as their top talent adjustment — while only about one in five companies has a mature governance model for autonomous agents. AI fluency teaches people how to prompt. It does not teach them to recognise a plausible-sounding wrong answer in a domain they do not know.
Steel-manning the other side: the law's approach has real critics. Legal scholars Johann Laux and Hannah Ruschemeier argue that the AI Act's focus on provider-side obligations does not adequately address the design and contextual factors that produce automation bias, and question whether merely mandating awareness of the bias does anything at all. They are probably right that awareness training is weak. That is an argument for changing the workflow, not for skipping the obligation.
Sequence Beats Explanation Quality
The most actionable finding in the paper is about ordering, and it costs nothing to implement. Presenting the AI's suggestion first led to worse outcomes when the AI was incorrect — for both groups, experts included. Show the answer before the human forms a view and you have anchored them, whatever their expertise.
The researchers' own recommendation follows from that: rather than leading with a detailed LLM explanation, require the user to commit to their own hypothesis first, then show the AI's suggestion as a set of alternatives to consider. The reviewer becomes a second opinion rather than a rubber stamp, and disagreement becomes a signal you can log and measure instead of a friction you have designed away.
This is a UI change and a workflow change. It is cheaper than any explainability platform, and unlike the platform, the study says it works.
MIT's Marzyeh Ghassemi frames the trade-off as a balance to be struck, not a problem to be solved: "Good AI systems can improve performance in some health settings, but this has to be balanced carefully with algorithmic deference that can lead to more error," she told MIT News.
Change the Workflow Before the Model
This Week: List every production AI workflow where the stated control is "a human reviews it." For each one, write down the actual job title and domain tenure of the person in that seat. If the reviewer is less expert than the person the model displaced, the review is not a control and your risk register is wrong. That inventory takes an afternoon and it is the whole exercise in miniature.
This Month: Flip the sequence on your two highest-consequence workflows. Capture the reviewer's judgement before the model's output is visible, then show the model's answer as an alternative. Log the disagreement rate. A rate near zero means you are measuring deference, not review — the same signal that shows up when an eval stops noticing a model has changed underneath it.
Before Renewal: Ask your explainability vendor for evidence that their explanations improve reviewer accuracy on wrong predictions, broken out by reviewer expertise. Not fidelity metrics, not attribution coverage — decision quality when the model is incorrect. Governance platforms like Credo AI and observability stacks like Arize AX can document what a model did; only your own logs can tell you whether the human caught it. Fund the disagreement-rate instrumentation before you renew the explanation layer.
Also this quarter: Route the outputs your reviewers disagree with, and the ones they approve fastest, to a domain expert for sampling. Fast approvals are where deference hides. This is the same operating discipline that Amazon adopted after AI-generated code caused outages and humans were made to sign off, and it is why a serious failure taxonomy puts process defects above hallucination.
The Bottom Line
Every automation wave has sold the same reassurance — the machine does the work, a person checks it — and every one has discovered that checking is a skill, not a step. Aviation learned it with autopilot and rebuilt cockpit procedure around it. Radiology learned it with computer-aided detection. Enterprise AI is learning it now, with the added twist that the explanation layer we bought to fix the problem is the thing making the least-qualified reviewer most confident.
The economics will keep pushing the review seat down the pay scale. The evidence says that seat is where your control silently stops working. And the two people who most need to hear this — the CFO approving the headcount model and the risk officer signing the oversight attestation — are usually in different meetings.
Explainability is not a safety control. It is a persuasion surface. Put an expert in front of it and measure how often they disagree. An expert who never disagrees is not oversight either.
Continue Reading
- Amazon's AI Code Created Outages. Now Humans Must Sign Off.
- TD Bank's First AI Agent: 15-Hour Reviews in 3 Minutes
- 92% Trust AI Code Scanning. 70% Have Vulns in Production.
- Group 1 Cut 700 Jobs. Better AI Wasn't the Reason.
- 10,000 AI Failures Exposed. Hallucination Isn't #1.
- AI Created a 62% Wage Gap Between Two Types of Workers
- DeepSeek Swapped the Model. Your Eval Didn't Notice.
- EU AI Transparency Law Starts Monday: Are You Ready?