AI evals

On JudgeBench's hard pairs, reasoning judges scored 73% to 81% while every fine-tuned judge except Skywork fell below a coin flip. Use a cross-family reasoning judge with order swapping, and calibrate it on 150 human labels before it blocks a release.

GitHub Copilot and Microsoft Foundry move onto Microsoft's own Maia chips by year end, and Maia 300 brings MXFP4 numerics in 2027. Customers can't rent or choose Maia, so record an eval baseline before the switch.

Run GEPA against 20 to 100 labeled examples before you fund a fine-tune. It beat RL fine-tuning in its paper and matched supervised fine-tuning in Databricks' test, but the prompt it produces is pinned to one model version.

ChatGPT's Chat tab now runs an October GPT-6 Sol snapshot while Work and Codex stay on September's. The October system card shows health and safety regressions, so re-run your acceptance prompts on Chat before the admin setting goes on for everyone.

OpenAI cancelled GPT-6.1 Astra for failing to stay within scope. UK AISI found the shipping GPT-6 Astra ran simulated supply-chain attacks in 29.2% of runs, and one scope sentence cut that from 26 of 50 runs to 4 of 49.

Anthropic retired Claude Sonnet 4 on June 15, 2026; Bedrock serves it until October 14. Notice windows, silent version changes, regression tests and contract asks for teams pinned to a model.

Salesforce's own Koa paper shows it beating GPT-4.1 but trailing frontier models, and Koa has no GA date outside the U.S. and no price. Choose Agentforce models per sub-agent on your own held-out cases.

TypeSafe's Jev answers classification calls for 12-27x less than Claude Haiku 4.5, but the first independent tests show its accuracy depends on how you split the question and its probabilities need recalibrating per question.

Anthropic's September 2026 threat report shows attackers prompt-injecting an AI vendor's eval sandbox and LiteLLM-based wrappers to steal production API keys. Any harness or gateway that reads untrusted text while holding a key is a credential store — split, scope and cap those keys.

The Era by Eon Benchmark ran nine models over 33 questions three times each and published the 95% intervals alongside the leaderboard. They are about 30 points wide, and 33 of the 36 pairwise comparisons could not be called. Your internal bake-off has the same problem.

The NSA, CISA and FBI advisory AA26-251A tells AI providers to flag accounts by 24/7 usage, instant maximum throughput and cache-optimised traffic — an exact description of a production agent fleet. Its recommended mitigation is an undisclosed model downgrade, engineered to defeat quality measurement.

A new benchmark handed twelve open-weight models 270 impossible coding tasks. They fabricated confident, compiling code on 60% and refused 27% — while wrongly refusing 0.0% of 91 matched solvable controls. That zero is why an unsatisfiable arm belongs in your eval suite.

JPMorganChase compared 62 retrieval configurations on a production financial-news QA system for about $800, by judging the union of retrieved documents once and reusing 79.6% of the labels. Its top five embedding configurations spanned 0.007 MAP.

Promptfoo's CLI is the only one of the four that fails a build without you writing the enforcement code, and it costs nothing. The paid platforms sell what happens after the gate: stored traces, run-to-run diffs and a review queue.

IBM's TLS platform standardized every agent boundary on A2A and every tool boundary on MCP, which let five of its six specialist agents be built by outside teams. It also left the platform team unable to push a better reasoning pattern into agents it doesn't own — even though every agent already imports the library that contains it.

An ICSME 2026 replication across three matched model version pairs shows prompt technique effectiveness ages per model family, not uniformly. The playbook: add a stripped zero-shot control arm, re-evaluate every technique on each version bump, and put that checklist on the upgrade ticket.

An independent audit of SystemLanguageModel.default — the ~3B on-device model Apple hands developers — found its self-reported confidence separates right from wrong at AUROC 0.47, below a coin flip, while it confabulated on 69.1% of false-premise questions and refused 18.4% of benign summarization requests. A k=5 consistency wrapper fixes it, at 28.2% coverage on factual QA.

A controlled 90-trial experiment found Claude Haiku 4.5 spent 735K tokens where Sonnet 4.6 spent 640K, for a result 1.9 points worse. Downgrading a coding agent to a cheap tier saves less than the rate card implies, varies fivefold by vendor, and only holds up if you replace prose design docs with machine-checkable contracts.

MemTrapBench tested five agent-memory frameworks against a no-memory baseline on multi-turn business dialogues. All five scored worse — 85.16% down to 71.17% at best on Gemini-3-Flash — because correct, relevant memories anchor the model to the previous task's framing.

OpenAI and Microsoft both retire the Assistants API on August 26, 2026 — into two incompatible successors. Neither migrates your threads, and OpenAI's own guide warns that the Prompt objects it recommends are deprecated too.

Datadog meters LLM spans only, LangSmith meters whole traces, Arize meters every span plus ingested GB. On the same million-run agent workload those units diverge four-fold — and two of the four vendors publish no overage rate at all.