If your feature says "only show this when the model is confident," you shipped a feature that shows everything. An independent audit of the on-device model Apple actually hands developers found that its self-reported confidence separates its correct answers from its wrong ones at AUROC 0.47 — statistically below a coin flip. The model reports that confidence at a mean of 98.8% almost regardless of whether it is right.
That is the whole finding, and it is not a story about hallucination rates. Every model gets things wrong. This one gets things wrong while emitting a number that looks exactly like the number it emits when it is correct, and there is no cheap way for your code to tell the difference at inference time.
What the Audit Actually Measured
The paper audited the configuration a developer can ship, not the one Apple benchmarks. "Confidently Wrong, Silently So: Auditing Undetectable Failures of a Deployed On-Device Language Model" was posted to arXiv on 24 August 2026 by Shashwat Pandey, Satwik Pandey and Suresh Raghu, one at UC Santa Cruz and two independent. It is a preprint and has not been peer-reviewed. Its target is SystemLanguageModel.default — described in the paper as "a general-purpose base model of roughly 3B parameters, queried on a commodity laptop" through the public FoundationModels framework.
That distinction is the point of the exercise. Apple's own reporting on the third generation of its foundation models covers AFM 3 Core and AFM 3 Core Advanced, published 8 June 2026, and grades them on human preference win rates — AFM 3 Core "earned preference on 45.6 percent of prompts compared to 23.3 percent for the 2025 baseline." A preference win rate tells you which output a rater liked better. It tells you nothing about whether the model knows when it is wrong.
The audit ran three red-team tasks: calibration on open-domain factual QA (n=2,900) and multi-step reasoning, confident confabulation on 110 false-premise questions with 150 answerable controls, and over-refusal across 310 summarization items spanning email, news, technical Q&A and ML news. Three open-weight models of comparable scale — Gemma-3-4B-it, Llama-3.2-3B-instruct and Ministral-3B — ran the identical frozen items through one deterministic grader. Note the asymmetry the authors flag themselves: those peers were served through a hosted API rather than on device, so "serving-pipeline differences cannot be fully excluded," and no equivalent guardrail stack sat in front of them. Read the peer numbers as context, not as a controlled comparison.
Why AUROC 0.47 Means the Confidence Field Is Dead Weight
AUROC measures whether a score ranks correct answers above wrong ones: 1.0 is perfect, 0.5 is a coin flip. Apple's model scored 0.47, with a confidence interval of 0.456 to 0.485 — an interval that does not include 0.5, so this is not a near-miss on chance. It is measurably below it, and not far enough below to be usefully inverted either. It is noise.
The second number is worse. Expected calibration error is the average gap between what a model claims and what it delivers, on the same 0–100 scale as the confidence itself. Apple's model posted an ECE of 70.1 — the worst among the four models tested. The mechanism is visible in the raw output: mean stated confidence of 98.8% under guided decoding and roughly 99% under free-form generation. The model has one setting.
This matters because verbalized confidence is the only uncertainty signal the framework hands you. There is no logprob API. If your product has a "we're not sure about this" state, a threshold on a prompted confidence number is almost certainly how you built it, and that branch has never fired.
No User-Visible Cue Separates a Right Answer From a Wrong One
The authors then asked the harder question: forget the model's own number, can anything on the surface of the output flag a confident error? They trained classifiers over 15 user-visible features — length, readability, hedging, specificity markers, stated confidence — to separate confident-correct from confident-wrong responses. Logistic regression reached AUROC 0.548; gradient boosting reached 0.566. The paper reports the result as equivalence-confirmed, meaning they statistically established indistinguishability rather than merely failing to find a difference.
Cheap single-generation probes did not rescue it either. Verbalized confidence scored 0.529 on TriviaQA and 0.519 on GSM8K. A hedge-to-verify ratio scored 0.543 and 0.590. Fusing the two got to 0.555 and 0.598. Response length was the best of the lot at 0.680 on GSM8K, and 0.513 on TriviaQA — which is to say it worked on arithmetic and not at all on facts.
Read that as an engineering constraint. Every downstream filter you might reach for — a regex for hedging language, a length heuristic, a UI treatment that dims low-confidence output, a second small model scoring the first one's phrasing — is operating on features that carry roughly no signal. You cannot post-process your way out of this. It is the same trap as trusting a scanner's clean report on AI-generated code that still ships vulnerabilities: the absence of a warning is not evidence.
The Guardrails Fail in Both Directions at Once
Apple's model confabulated on 69.1% of false-premise questions, against 26.4% for Gemma, 12.7% for Llama and 7.3% for Ministral. The paper defines confident confabulation precisely: the model "supplies a substantive answer at verbalized confidence ≥80" to a question built on a premise that is false. Roughly seven times in ten, asked something with no true answer, it invented one and stamped it as certain.
Then it went the other way. On the 310 benign summarization items, Apple's model refused 18.4% — 57 of 310. Gemma, Llama and Ministral each refused 0.0% — though a bare hosted model with no safety layer in front of it refusing nothing is close to tautological, so the number that matters here is Apple's own.
That asymmetry is the finding that should reach your product review, because summarization is not an edge case. It is the headline use case in Apple's own developer guidance, and it is what the shipping apps built on the framework — SmartGym, Stoic, Signeasy, Agenda and OmniFocus 4 among the launch cohort — are largely doing. Nearly one benign summarization request in five came back as a refusal, while two false-premise questions in three came back as confident fiction. The same guardrail stack is simultaneously too tight and too loose, on different tasks, with no signal telling you which mode you are in.
The Fix Works, and Then It Answers 28 Questions in 100
The one mitigation that worked needs no model access, no fine-tuning and no vendor cooperation. Draw k=5 independent samples for the same query and abstain when the final answers disagree. The paper's phrasing: "for each query we draw k stochastic samples and abstain (or escalate) when their final answers disagree, converting undetectable confident errors into explicit abstentions."
The results are large, and they come off small item sets. On the false-premise questions, confident confabulation fell from 75.0% to 2.6% (n=76). On factual QA, selective accuracy — accuracy over the answers the system chooses to give — rose from 42.7% to 82.8% and ECE collapsed from 57.2 to 17.2 (n=103).
Now the sentence everyone skips. That selective accuracy is measured at 28.2% coverage. On factual QA, the system answers 28 queries in 100 and abstains on the other 72.
Scope that number before you budget against it. It is one task at one threshold, tuned on held-out items — the authors are explicit that the mitigation is "demonstrated at a single tuned operating point rather than swept across all deployment budgets" — and the wrapper was never run on the summarization set at all. It may not even port cleanly. The wrapper abstains when "their final answers disagree," which is well defined for a factual answer and not remotely obvious for two free-text summaries. If your traffic is mostly summarization you need your own coverage number and your own definition of disagreement, which is the whole reason step 4 below exists.
The compute is genuinely not the problem. Apple ships this inference free of cost, on hardware you do not own or pay for, and the paper notes that "because the samples are independent, on-device batching keeps wall-clock latency below the nominal k× factor." Five samples on a 3-billion-parameter model quantized to 2 bits costs you battery and some milliseconds. Fine.
The problem is the abstained remainder, whatever it turns out to be on your traffic. Those queries do not disappear. They go somewhere, and where they go is a decision you have almost certainly not made yet.
Your Abstention Path Is a Data-Residency Decision
Every query the wrapper declines has to be answered by something else, and the something else is off the device. That is the exact review you skipped when you chose on-device inference.
Apple's answer is PrivateCloudComputeLanguageModel. The WWDC26 framework update introduced a LanguageModel protocol so local and server models can back the same LanguageModelSession, exposed a 32,000-token PCC model with reasoning levels against an on-device context window of 8,192 tokens, and opened the abstraction to third-party Swift packages from Anthropic and Google. On paper the escalation path is one line of code.
Read the eligibility terms before you plan around it. Apple's Private Cloud Compute developer page restricts free PCC access to developers who "are enrolled in the App Store Small Business Program" and "have fewer than 2 million first-time app downloads from any of their apps on the App Store." And then: "If any app subsequently exceeds the 2 million first-time downloads threshold, or the developer is no longer enrolled in the App Store Small Business Program, the developer will be notified and must migrate to an alternative solution within 6 months."
If you are the kind of company that has a data-residency review, you are the kind of company that trips that threshold. There is no paid PCC tier. Success on the App Store is the event that terminates your privacy-preserving fallback and starts a six-month clock.
So the honest architecture is split: some share of queries answered on-device for free, the rest routed to a commercial API under a real contract. Price it. At Gemini 2.5 Flash-Lite's published paid rate of $0.10 per million input tokens and $0.40 per million output, a feature doing a million queries a month at roughly 800 input and 200 output tokens each, escalating at the paper's 71.8% rate, sends about 718,000 of them to the cloud — call it $115 a month by my arithmetic on those rates. That is nothing.
The bill was never the reason you went on-device. The reason was that user content stayed on the handset and your security team did not need a data processing agreement, a sub-processor disclosure or an EU transfer mechanism. A material abstention rate hands all three of those back, and "material" starts a long way below 72%. The same accounting applies to any locally-run agent model whose escalations leave no trace in a gateway log, and it is the same calculus as choosing an alternative provider for data-sensitive workloads — the residency question follows the escalation, not the primary path.
Apple Told You Not to Do This, and That Only Half Excuses It
Here is the strongest version of the counter-argument, and it is stronger than most coverage will admit.
Apple's guidance is unusually blunt. Its WWDC session on prompt design and safety states: "Avoid relying on the system language model for facts. If you do want to generate facts, consider providing the model with verified information written into your prompt." And: "For knowledge the model doesn't know, it may hallucinate, which is the technical word for completely making up an answer." The framework session adds that the model "is not designed for world knowledge or advanced reasoning, which are tasks you might typically use server-scale LLMs for."
By that standard, a 69.1% confabulation rate on false-premise trivia measures the model outside its documented envelope. Apple can fairly say it told you.
Three things survive that defence.
First, the calibration failure is not confined to trivia. AUROC 0.47 and ECE 70.1 are properties of the confidence signal itself, and the single-generation probes were just as useless on multi-step reasoning as on facts. The model is uninformative about its own reliability everywhere it was measured.
Second, the 18.4% over-refusal is squarely inside the documented envelope. Summarization is the flagship use case. Nothing in the guidance prepares you for one benign summary request in five bouncing.
Third, "don't use it for facts" is not a control you can enforce. You do not choose what users type into a text field. A note-taking app with a summarize button will be handed a note containing a question, and the model will answer it at 99% confidence. The guidance describes an intent; the audit measures what the system does when reality does not cooperate.
There is also a real caveat running the other way, and the authors state it themselves: "the underlying build and operating-system version are not recorded in our generation logs," and they commit to pinning the exact build for the camera-ready. That matters, because Apple says it already made "adjustments in iOS 26.4 to reduce the number of false positives" and is "continuing to make even more improvements in iOS 27," which ships a new on-device model "rebuilt from the ground up" — and false positives are precisely the over-refusal number. The 18.4% may already be stale.
The calibration result is the one that should worry you anyway, and nobody has published a replacement figure for the new build. The authors released their code and frozen evaluation items so you can run the protocol yourself. That is the correct response to a stale number: re-measure it, on your build, with your prompts. Which is the same conclusion as refusing to inherit a vendor's own safety evaluations — a vendor's assurance is an input to your testing, not a substitute for it.
Do This Before Your Next Release
This Week:
- Grep for the threshold. Search your codebase for any comparison against a model-reported confidence value — a prompted
confidencefield, a "certainty" score, anything scoring 0–100. Delete the branch or make it unconditional. At AUROC 0.47 that condition is a random gate on user-visible output, and today it is passing everything. - Split your call volume by task. Count what share of your on-device calls are summarization, extraction and classification versus anything that resolves to a factual lookup. The first group is what the model is built for. The second is where the 69% lives, and you probably have more of it than you think, because users put questions inside documents.
- Look up your first-time download count. If any app is near two million, the free PCC tier is a six-month asset, not an architecture. Find that number before you design around it.
This Month:
- Build the k=5 wrapper behind a flag and measure your own coverage. 28.2% is the paper's number on the paper's factual-QA items, and the wrapper was never measured on summarization at all. Yours will differ, and the only figure that matters is the one from your prompts on your build. Instrument the disagreement rate before you instrument anything else.
- Write the fallback contract now. Name the model, the region, the retention terms and the DPA that covers whatever share you end up escalating. If the answer is "we'd send it to a frontier API," that is a sub-processor disclosure and, in the EU, a transfer mechanism. Get your privacy counsel the question this month, not the week before launch.
- Put abstention rate on the dashboard next to error rate. A consistency wrapper converts silent wrongness into visible refusal, and refusal is a product metric with a cost. Track it in Langfuse or Braintrust alongside the accuracy number, and set an alert on it — a rising abstention rate after an OS update is your only early warning that the model underneath you changed.
Before You Ship:
- Run the released audit on your own build. Record the OS version in your logs — the thing the authors could not do. On-device models update silently with the operating system, and you have no release notes, no version pin and no deprecation window. Your regression suite is the only version control you get.
- Grade pass/fail, not average. An 82.8% selective accuracy at 28.2% coverage and a 42.7% accuracy at full coverage can produce similar-looking aggregate scores and completely different products. This is the same measurement error that makes an average agent score of 73 hide a 30% usability rate, and the same reason a memory-augmented agent has to be tested with memory switched off.
The Bottom Line
The industry spent two years learning that models hallucinate, and built a generation of products on the assumption that a model at least knows when it is guessing. This audit says that for one model shipping to hundreds of millions of devices, that assumption is worth AUROC 0.47 — and that no surface feature, no hedging heuristic and no length filter recovers it.
The fix is old, cheap and completely available to you: ask five times and abstain when the answers disagree. It works. On the paper's factual-QA items it also turned a 100%-coverage feature into a 28%-coverage one. Coverage is a threshold you can move — but the paper published a single operating point and not the curve, so you will be discovering your own. Wherever you land, it is a different product, with a cloud dependency, a contract and a residency review attached.
That is the trade, and it is fine to take it. What is not fine is shipping the first version and believing you took it, because the confidence number said 99%.
A model that is always sure is not a model with high confidence. It is a model with no confidence signal at all.
Continue Reading
Apple Runs a 20B AI Model on iPhone. Your Data Never Leaves.
iOS 27 Lets Employees Pick Their AI. Is Your MDM Ready?
Meta's Agent Model Fits on a Laptop. Nothing Logs It.
Anthropic's Evals Maxed Out. Stop Inheriting Its Assurance.
Reasoning Trap: Smarter AI Agents Hallucinate More Tools
10,000 AI Failures Exposed. Hallucination Isn't #1.