AI evals

Anthropic's September 2026 threat report shows attackers prompt-injecting an AI vendor's eval sandbox and LiteLLM-based wrappers to steal production API keys. Any harness or gateway that reads untrusted text while holding a key is a credential store — split, scope and cap those keys.

The Era by Eon Benchmark ran nine models over 33 questions three times each and published the 95% intervals alongside the leaderboard. They are about 30 points wide, and 33 of the 36 pairwise comparisons could not be called. Your internal bake-off has the same problem.

The NSA, CISA and FBI advisory AA26-251A tells AI providers to flag accounts by 24/7 usage, instant maximum throughput and cache-optimised traffic — an exact description of a production agent fleet. Its recommended mitigation is an undisclosed model downgrade, engineered to defeat quality measurement.

A new benchmark handed twelve open-weight models 270 impossible coding tasks. They fabricated confident, compiling code on 60% and refused 27% — while wrongly refusing 0.0% of 91 matched solvable controls. That zero is why an unsatisfiable arm belongs in your eval suite.

JPMorganChase compared 62 retrieval configurations on a production financial-news QA system for about $800, by judging the union of retrieved documents once and reusing 79.6% of the labels. Its top five embedding configurations spanned 0.007 MAP.

Promptfoo's CLI is the only one of the four that fails a build without you writing the enforcement code, and it costs nothing. The paid platforms sell what happens after the gate: stored traces, run-to-run diffs and a review queue.

IBM's TLS platform standardized every agent boundary on A2A and every tool boundary on MCP, which let five of its six specialist agents be built by outside teams. It also left the platform team unable to push a better reasoning pattern into agents it doesn't own — even though every agent already imports the library that contains it.

An ICSME 2026 replication across three matched model version pairs shows prompt technique effectiveness ages per model family, not uniformly. The playbook: add a stripped zero-shot control arm, re-evaluate every technique on each version bump, and put that checklist on the upgrade ticket.

An independent audit of SystemLanguageModel.default — the ~3B on-device model Apple hands developers — found its self-reported confidence separates right from wrong at AUROC 0.47, below a coin flip, while it confabulated on 69.1% of false-premise questions and refused 18.4% of benign summarization requests. A k=5 consistency wrapper fixes it, at 28.2% coverage on factual QA.

A controlled 90-trial experiment found Claude Haiku 4.5 spent 735K tokens where Sonnet 4.6 spent 640K, for a result 1.9 points worse. Downgrading a coding agent to a cheap tier saves less than the rate card implies, varies fivefold by vendor, and only holds up if you replace prose design docs with machine-checkable contracts.

MemTrapBench tested five agent-memory frameworks against a no-memory baseline on multi-turn business dialogues. All five scored worse — 85.16% down to 71.17% at best on Gemini-3-Flash — because correct, relevant memories anchor the model to the previous task's framing.

OpenAI and Microsoft both retire the Assistants API on August 26, 2026 — into two incompatible successors. Neither migrates your threads, and OpenAI's own guide warns that the Prompt objects it recommends are deprecated too.

Datadog meters LLM spans only, LangSmith meters whole traces, Arize meters every span plus ingested GB. On the same million-run agent workload those units diverge four-fold — and two of the four vendors publish no overage rate at all.