agent evaluation

Beacon Software acquired AI red-teaming firm Haize Labs to run its internal AI platform, and the announcement says nothing about existing customers. Here is what to secure before renewal.

On UNICEF's own statistics, a generic SDMX MCP server scored 0.074 at returning the right figure for the right year, below the 0.147 of no tools. A purpose-built server scored 0.990. As the UN opens 44 million data points to agents, the lesson for enterprise data teams is to fund the resolver, not the connector.

On Specific Labs' Real-SWE benchmark of licensed private codebases, Fable 5.1 resolved the most tasks but cost about $17.94 per fix to Gemini 3.8 Flash's $8.01 — and each model fails in its own way, so your review gate should follow the agent you pick.

An unrepaired probability sample of 400 MCP servers found only 48.8% complete an initialize handshake, and 37.5% never start at all. Meanwhile 68.8% of BFCL v4's extracted tool definitions and 85.6% of UltraTool's are exact repeats, against 0.4% for real MCP tools.

The Era by Eon Benchmark ran nine models over 33 questions three times each and published the 95% intervals alongside the leaderboard. They are about 30 points wide, and 33 of the 36 pairwise comparisons could not be called. Your internal bake-off has the same problem.

ERPBench ran 100 identical ERP problems through six model families twice — solo against fixed opponents, then all six competing in one market. DeepSeek won the first, Gemini the second, and the two agreed on 21 of 100 tasks.

SWE-Gate scored review-constraint compliance separately from functional tests across 303 repository-level repair tasks. Of 644 agent patches that passed the tests, 221 violated constraints taken from the original pull request reviews.

A new benchmark handed twelve open-weight models 270 impossible coding tasks. They fabricated confident, compiling code on 60% and refused 27% — while wrongly refusing 0.0% of 91 matched solvable controls. That zero is why an unsatisfiable arm belongs in your eval suite.

A study of 14,922 agent trajectories across five coding benchmarks finds Claude succeeds by matching the human patch's file scope while Qwen succeeds by exceeding it at every scale. Equal pass rates buy unequal diffs, and your ticket template is the control.

MemTrapBench tested five agent-memory frameworks against a no-memory baseline on multi-turn business dialogues. All five scored worse — 85.16% down to 71.17% at best on Gemini-3-Flash — because correct, relevant memories anchor the model to the previous task's framing.

StartupBench ran nine frontier agents across 97 workflows taken from products AI startups already sell. A 73.67 rubric average produced an acceptable deliverable on 29.55% of runs — and agents satisfied auxiliary requirements more often than core ones.

Alibaba says Qwen3.8-Max coded unattended for 16 days, and unlike every rival long-horizon claim it shipped the whole commit trace on public GitHub. Open it and the 16 days become 'more than ten' in Alibaba's own words, 13 of 648 commits turn out to be a human's, and one person still holds the merge keys.