vendor benchmarks

Microsoft's Decision-1 lists at $0.042 per million input tokens with free output, the same price as TypeSafe's Jev. Its 'highest accuracy' claim rests on 36 unnamed benchmarks with no published scores or calibration data.

A vendor-run benchmark is a screening signal. Business cases should cite independent runs with per-task logs, led by Epoch AI and Artificial Analysis, plus a 50-task replication on your own data.

Reflection's Beam is a 501B Apache 2.0 open model from a US lab, and its own table has it losing to Kimi K3 on 13 of 14 shared benchmarks. Weights and pricing aren't out yet.

Inception's Mercury Voice cuts the voice-agent LLM line to about $0.009 a minute, but its 320ms latency is model-only and its benchmark win arrives without numbers. Bake it off on your own calls before you swap.

NVIDIA's Kumo Tabular ships under OpenMDW-1.1 for commercial use, while TabPFN-3.5's open weights stay non-commercial. Its #1 TabArena Elo is NVIDIA's own run, and it was pretrained only to 60,000 rows and 100 columns.

GPT-6.1 Sol costs a fifth of GPT-6 Astra per token and ties it on coding, but trails it by 11 points on Terminal-Bench Science. It still wins on cost per solved task — if your failures are cheap.

Claude Sonnet 5.5 lists at half Opus 5.5's token price, but Artificial Analysis measured it at $7.60 per task at max effort against $3.46 for Opus at xhigh for the same score. It saves money only at low and high effort, and five breaking changes sit behind the model-ID swap.

GPT-6 Sol costs $2/$10 per million tokens, half GPT-5.6 Sol's promo rate, permanently. But OpenAI's headline evals ran at xhigh or max effort, and only one carries a dollar cost per task.

OpenAI scored Astra for Law at 54% on Vals AI's private validation set, against its own GPT-6 Astra. On Vals' public leaderboard, three general frontier models already sit at 55.29% and Astra for Law is not listed.

Salesforce AI Research and UIUC replaced the scoring pass in KV cache eviction with a uniform random draw, kept the prompt pinned, and matched the strongest published evictor across four models and six reasoning tasks — at 32-43% higher vLLM throughput. Random-plus-prompt-protection is now the baseline any 'intelligent cache compression' claim has to beat.

Nvidia's 3,400 tokens/sec on Groq 3 LPX and Cerebras' 4,400 on CS-4 are both single-stream figures. On a 256-LPU rack at 100K context, the memory math caps concurrency near a batch of 12 — so any capacity plan sized off a headline token rate is sized for one user.

Alibaba says Qwen3.8-Max coded unattended for 16 days, and unlike every rival long-horizon claim it shipped the whole commit trace on public GitHub. Open it and the 16 days become 'more than ten' in Alibaba's own words, 13 of 648 commits turn out to be a human's, and one person still holds the merge keys.