Defeating Nondeterminism in LLM Inference
by Thinking Machines Lab (Horace He)
Why temperature 0 still gives different answers, and how batch-invariant kernels make vLLM inference bitwise reproducible.
Overview
Defeating Nondeterminism in LLM Inference is a long-form essay by Horace He with colleagues at Thinking Machines Lab, published September 10, 2025, with a citable DOI (10.64434/tml.20250910). It starts with the observation that temperature-0 sampling is not reproducible on APIs or on self-hosted vLLM and SGLang. It then argues that the usual explanation, floating-point non-associativity plus concurrency, is incomplete: a typical LLM forward pass uses no atomic adds and is run-to-run deterministic. The real cause is that kernels compute different reduction orders depending on batch size, and batch size depends on server load. The middle sections show how to make each operation batch-invariant: one batch row per core for RMSNorm, a single tiling configuration instead of Split-K or stream-K for matmul (about 20% slower than cuBLAS), and fixed-size Split-KV splits plus a KV cache updated before the attention kernel. The experiments use Qwen3-235B-A22B-Instruct-2507: 1,000 temperature-0 completions produced 80 unique outputs, diverging at token 103, and all 1,000 were identical with the batch-invariant kernels. On Qwen3-8B the deterministic path took 42 seconds against vLLM's 26 after an improved attention kernel. A final RL experiment shows that bitwise-matching sampler and trainer keep KL divergence at 0. The code is in the MIT-licensed batch_invariant_ops repo, which swaps in PyTorch kernels via torch.Library. The post is from September 2025, slightly over a year old, and vLLM's deterministic mode builds on it.
At a Glance
- Topic
- ML
- Level
- Advanced
- Format
- Guide
- Cost
- Free
- Duration
- ~30-40 min read, plus optional time with the batch_invariant_ops code
- Provider
- Thinking Machines Lab (Horace He)
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Explain why floating-point non-associativity alone does not make an LLM forward pass nondeterministic
- ✓Identify batch-size dependence as the source of nondeterminism in served LLM outputs
- ✓Write batch-invariant RMSNorm by keeping each batch row's reduction on a single core
- ✓Choose matmul tiling that avoids Split-K and stream-K so results do not change with batch size
- ✓Make attention batch-invariant under chunked prefill and prefix caching using fixed-size Split-KV
- ✓Use torch.Library and the batch_invariant_ops package to run deterministic inference in vLLM
- ✓Measure the throughput cost of deterministic inference against vLLM's default kernels
- ✓Apply bitwise-identical sampling and training to get true on-policy reinforcement learning
Highlights
- •Reproducible numbers: 80 unique completions out of 1,000 at temperature 0 on Qwen3-235B, reduced to 1 with the fix
- •Ships working code in batch_invariant_ops (about 1.1k GitHub stars, MIT license) with a vLLM proof of concept
- •Reached 345 points and 130 comments on Hacker News when published
- •Connects a serving detail to RL training stability, showing reward collapse near step 318 without the correction
- •Gives honest performance costs (about 20% on matmul, 42s against 26s end to end) instead of claiming the fix is free
Who It’s For
Best For
- ✓Inference engineers running vLLM or SGLang who need reproducible outputs for tests or audits
- ✓Post-training and RL engineers whose sampler and trainer numerics disagree
- ✓GPU kernel developers learning how reduction order affects correctness
- ✓Teams building evals that must give the same answer twice
Prerequisites
- •Comfort with PyTorch and how GPU kernels parallelize reductions
- •Working knowledge of LLM serving concepts such as batching, KV cache, chunked prefill and FlashAttention
FAQ
What is Defeating Nondeterminism in LLM Inference?
A free technical essay by Horace He of Thinking Machines Lab for inference and post-training engineers. It shows that LLM nondeterminism comes from kernels whose results change with batch size, not from concurrency alone, and explains how to write batch-invariant RMSNorm, matmul and attention so vLLM returns identical outputs and RL training becomes truly on-policy.
Is Defeating Nondeterminism in LLM Inference free?
Defeating Nondeterminism in LLM Inference is free to access.
What level is Defeating Nondeterminism in LLM Inference for?
Defeating Nondeterminism in LLM Inference is aimed at a advanced audience. Recommended background: Comfort with PyTorch and how GPU kernels parallelize reductions, Working knowledge of LLM serving concepts such as batching, KV cache, chunked prefill and FlashAttention.
How long does Defeating Nondeterminism in LLM Inference take?
Expect roughly ~30-40 min read, plus optional time with the batch_invariant_ops code. Most learners work through it at their own pace.
What will I learn from Defeating Nondeterminism in LLM Inference?
You'll learn: Explain why floating-point non-associativity alone does not make an LLM forward pass nondeterministic; Identify batch-size dependence as the source of nondeterminism in served LLM outputs; Write batch-invariant RMSNorm by keeping each batch row's reduction on a single core; Choose matmul tiling that avoids Split-K and stream-K so results do not change with batch size; Make attention batch-invariant under chunked prefill and prefix caching using fixed-size Split-KV; Use torch.Library and the batch_invariant_ops package to run deterministic inference in vLLM; Measure the throughput cost of deterministic inference against vLLM's default kernels; Apply bitwise-identical sampling and training to get true on-policy reinforcement learning.
Topics
Sources
This page was written from 3 sources, 2 on domains other than thinkingmachines.ai.