self-hosted inference

Mistral Large 4's 1.05T-parameter weights are on Hugging Face. At FP8 they fit one eight-GPU DGX B200 node, internal use is free, and serving the model to outside customers needs a separate commercial licence from Mistral.

NVIDIA AI Enterprise lists at $4,500 per GPU per year. For bare-metal inference, vLLM or SGLang on your own stack, or Red Hat AI for a support contract, replaces it; vGPU and production NIM are where you still need it.

KServe and llm-d stack: KServe's LLMInferenceService runs llm-d underneath. Ray Serve replaces both, and it suits Ray-native Python teams more than Kubernetes platform teams.

vLLM's October advisories let one request crash or silently hang a shared inference engine, and two paths lead to code execution. Upgrade to 0.32.0, update Pillow, remove old Ghostscript and fix your health probes.

An H100 running gpt-oss-120b at Lambda's $4.29 an hour must sustain 5,958 tokens per second every hour of the month to match a $0.20 blended API rate. Business-hours traffic leaves it at about twice the API's cost.

Meta's 816B Muse Spark Open is free for internal use, but its license bans operating critical infrastructure and any nuclear or military use, the same clauses Llama 3 carried.

Prime Intellect bought Moondream for its Photon engine, and Moondream Cloud and Lens close November 13. Base weights self-host free, but local fine-tunes still need a Moondream API key.

Intel's air-cooled D-series GPU holds a 300B model on one PCIe card, but at 1.5 TB/s a large dense model streams single-digit tokens per second. Sort your workloads by capacity versus bandwidth before the quote arrives.

Run FP8 by default. For 4-bit, INT4-AWQ beat NVFP4 on speed on three of four models in an independent Blackwell benchmark, and NVIDIA's own NVFP4 Gemma file was nearly the size of FP8.

Mistral Large 4 beats Kimi K3 on Harvey's legal agent benchmark and trails it on the Vals Index. It is a discounted preview API for now, and the licence for its end-of-October weights is unpublished.

Reflection's Beam is a 501B Apache 2.0 open model from a US lab, and its own table has it losing to Kimi K3 on 13 of 14 shared benchmarks. Weights and pricing aren't out yet.

For air-gapped LLMs, run Apache-2.0 weights like gpt-oss-120b on vLLM, via Red Hat AI if you need a vendor. NVIDIA AI Enterprise's per-GPU fee is the loser, and an 8-GPU air-gapped node costs 30-40x the hosted API at 1B tokens a month, and even two GPUs cost about 9x.

Salesforce AI Research and UIUC replaced the scoring pass in KV cache eviction with a uniform random draw, kept the prompt pinned, and matched the strongest published evictor across four models and six reasoning tasks — at 32-43% higher vLLM throughput. Random-plus-prompt-protection is now the baseline any 'intelligent cache compression' claim has to beat.

Tencent's Hy4 preview markets 49B active parameters, but all 770B must stay resident — roughly 780GB in FP8, which an 8xH100 node cannot load at all. Size a Mixture-of-Experts model on total parameters and price throughput on active ones.

Nvidia carries $279B in supply commitments, an increase it attributes primarily to memory, and still guided gross margin from 75% down to a 71-72% trough on RAM costs. Your server quote has no such contract behind it, and every self-hosted inference TCO model built on 2025 hardware prices is now understated.

A new arXiv paper measures the corrective value of a failed tool call in an agent transcript and finds it negative on every small model tested: repeat probability rises from 0.06 to 0.54. Clean-context retry was the worst harness tested; paraphrasing the failure removed 76% of the effect for free.

TensorRT-LLM 1.2 removed the TensorRT engine build entirely, invalidating most published advice on this choice. vLLM is the default, SGLang wins prefix-heavy agent traffic, and the properties that actually matter never appear in a benchmark.

Nvidia's cross-model KV cache mapper moves a 32,768-token cache in 277.6ms instead of 6,975.3ms of re-prefill. Broken out by benchmark, GSM8K survived on one of six model pairs — Llama 3.1 8B to 70B fell from 81.12 to 14.78.

Seven routes off OpenAI, priced and deployment-matched on 12 August 2026. Residency is already solved almost everywhere — only Mistral, Cohere private deployment and self-hosted open weights clear a jurisdiction rule.

One of the first multi-vendor serving benchmarks on Kimi K3 puts AMD's MI355X at roughly 1.5x Nvidia B300's tokens per dollar — but that rests on a $2.50/GPU-hour rate you cannot currently buy. At the only purchasable on-demand rates, the gap narrows to 6%.