self-hosted inference

Salesforce AI Research and UIUC replaced the scoring pass in KV cache eviction with a uniform random draw, kept the prompt pinned, and matched the strongest published evictor across four models and six reasoning tasks — at 32-43% higher vLLM throughput. Random-plus-prompt-protection is now the baseline any 'intelligent cache compression' claim has to beat.

Tencent's Hy4 preview markets 49B active parameters, but all 770B must stay resident — roughly 780GB in FP8, which an 8xH100 node cannot load at all. Size a Mixture-of-Experts model on total parameters and price throughput on active ones.

Nvidia carries $279B in supply commitments, an increase it attributes primarily to memory, and still guided gross margin from 75% down to a 71-72% trough on RAM costs. Your server quote has no such contract behind it, and every self-hosted inference TCO model built on 2025 hardware prices is now understated.

A new arXiv paper measures the corrective value of a failed tool call in an agent transcript and finds it negative on every small model tested: repeat probability rises from 0.06 to 0.54. Clean-context retry was the worst harness tested; paraphrasing the failure removed 76% of the effect for free.

TensorRT-LLM 1.2 removed the TensorRT engine build entirely, invalidating most published advice on this choice. vLLM is the default, SGLang wins prefix-heavy agent traffic, and the properties that actually matter never appear in a benchmark.

Nvidia's cross-model KV cache mapper moves a 32,768-token cache in 277.6ms instead of 6,975.3ms of re-prefill. Broken out by benchmark, GSM8K survived on one of six model pairs — Llama 3.1 8B to 70B fell from 81.12 to 14.78.

Seven routes off OpenAI, priced and deployment-matched on 12 August 2026. Residency is already solved almost everywhere — only Mistral, Cohere private deployment and self-hosted open weights clear a jurisdiction rule.

One of the first multi-vendor serving benchmarks on Kimi K3 puts AMD's MI355X at roughly 1.5x Nvidia B300's tokens per dollar — but that rests on a $2.50/GPU-hour rate you cannot currently buy. At the only purchasable on-demand rates, the gap narrows to 6%.