GPU capacity planning

S&P says there is no obvious insurance product for the GPUs inside AI data centers, and colo agreements often cap the operator's liability at three months of fees. If you own a GPU cluster in someone else's hall, the gap between that cap and the hardware is yours to close.

Salesforce AI Research and UIUC replaced the scoring pass in KV cache eviction with a uniform random draw, kept the prompt pinned, and matched the strongest published evictor across four models and six reasoning tasks — at 32-43% higher vLLM throughput. Random-plus-prompt-protection is now the baseline any 'intelligent cache compression' claim has to beat.

Tencent's Hy4 preview markets 49B active parameters, but all 770B must stay resident — roughly 780GB in FP8, which an 8xH100 node cannot load at all. Size a Mixture-of-Experts model on total parameters and price throughput on active ones.

Nvidia's 3,400 tokens/sec on Groq 3 LPX and Cerebras' 4,400 on CS-4 are both single-stream figures. On a 256-LPU rack at 100K context, the memory math caps concurrency near a batch of 12 — so any capacity plan sized off a headline token rate is sized for one user.

TensorRT-LLM 1.2 removed the TensorRT engine build entirely, invalidating most published advice on this choice. vLLM is the default, SGLang wins prefix-heavy agent traffic, and the properties that actually matter never appear in a benchmark.