NVIDIA AI EnterpriseRun vLLM on Bare Metal to Skip NVIDIA AI Enterprise Fees
NVIDIA AI Enterprise lists at $4,500 per GPU per year. For bare-metal inference, vLLM or SGLang on your own stack, or Red Hat AI for a support contract, replaces it; vGPU and production NIM are where you still need it.
October 29, 2026 · 14 min readllm-dKServe and llm-d Beat Ray Serve for Kubernetes Teams
KServe and llm-d stack: KServe's LLMInferenceService runs llm-d underneath. Ray Serve replaces both, and it suits Ray-native Python teams more than Kubernetes platform teams.
October 26, 2026 · 14 min readvLLMvLLM 0.30.0 Can Run Code Through Ghostscript Images
vLLM's October advisories let one request crash or silently hang a shared inference engine, and two paths lead to code execution. Upgrade to 0.32.0, update Pillow, remove old Ghostscript and fix your health probes.
October 26, 2026 · 10 min readself-hosted inferenceTogether AI and Bedrock beat self-hosted gpt-oss-120b on cost
An H100 running gpt-oss-120b at Lambda's $4.29 an hour must sustain 5,958 tokens per second every hour of the month to match a $0.20 blended API rate. Business-hours traffic leaves it at about twice the API's cost.
October 23, 2026 · 16 min readstructured outputsEvery Structured Output Engine Still Needs Field-Level Validation
Strict JSON modes guarantee the payload's shape and nothing about its values. What separates the engines is what they do with a schema keyword they can't enforce: Claude's API and llguidance reject it, while XGrammar and llama.cpp can let it through.
October 14, 2026 · 14 min readair-gapped deploymentBest Air-Gapped LLM Stack: Apache Weights on vLLM Beat NVIDIA's Fee
For air-gapped LLMs, run Apache-2.0 weights like gpt-oss-120b on vLLM, via Red Hat AI if you need a vendor. NVIDIA AI Enterprise's per-GPU fee is the loser, and an 8-GPU air-gapped node costs 30-40x the hosted API at 1B tokens a month, and even two GPUs cost about 9x.
September 28, 2026 · 14 min readKV cacheRandom Eviction Matched the Scorers. Protect the Prompt.
Salesforce AI Research and UIUC replaced the scoring pass in KV cache eviction with a uniform random draw, kept the prompt pinned, and matched the strongest published evictor across four models and six reasoning tasks — at 32-43% higher vLLM throughput. Random-plus-prompt-protection is now the baseline any 'intelligent cache compression' claim has to beat.
September 9, 2026 · 13 min readMixture of ExpertsTencent Says 49B Active. Your Node Loads All 770B.
Tencent's Hy4 preview markets 49B active parameters, but all 770B must stay resident — roughly 780GB in FP8, which an 8xH100 node cannot load at all. Size a Mixture-of-Experts model on total parameters and price throughput on active ones.
August 30, 2026 · 12 min readOptimumNvidia Buys the Bridge to Trainium. Go Grep Your Imports.
Nvidia's reported $12.9B deal for Hugging Face puts optimum-neuron, optimum-habana, optimum-intel and optimum-amd — the Transformers bridge for every rival accelerator — inside Nvidia. optimum-tpu was archived read-only in January, and its own README points the exit at vLLM.
August 27, 2026 · 10 min readagent harnessWiping the Transcript Was the Worst Retry. Paraphrase.
A new arXiv paper measures the corrective value of a failed tool call in an agent transcript and finds it negative on every small model tested: repeat probability rises from 0.06 to 0.54. Clean-context retry was the worst harness tested; paraphrasing the failure removed 76% of the effect for free.
August 26, 2026 · 13 min readvLLMvLLM vs TensorRT-LLM vs SGLang: Default to vLLM
TensorRT-LLM 1.2 removed the TensorRT engine build entirely, invalidating most published advice on this choice. vLLM is the default, SGLang wins prefix-heavy agent traffic, and the properties that actually matter never appear in a benchmark.
August 24, 2026 · 18 min read