Apple's On-Device Model Is 99% Sure. Sample It Five Times.
An independent audit of SystemLanguageModel.default — the ~3B on-device model Apple hands developers — found its self-reported confidence separates right from wrong at AUROC 0.47, below a coin flip, while it confabulated on 69.1% of false-premise questions and refused 18.4% of benign summarization requests. A k=5 consistency wrapper fixes it, at 28.2% coverage on factual QA.
August 25, 2026 · 15 min read