If you sized an inference cluster off a vendor's headline tokens-per-second number, you sized it for one user. Nvidia says its Groq 3 LPX rack hit 3,400 output tokens per second on Gemma 4 31B at a 100,000-token context. Cerebras says its CS-4 does more than 4,400 tokens per second per user on GPT-OSS-120B. Both figures are almost certainly real. Both describe a machine serving a single request at a time.
The number your capacity plan needs is a different one: how many requests that hardware holds in flight at your context length before it runs out of memory. On a fully populated 256-LPU LPX rack with every request carrying 100K input tokens, The Register worked the arithmetic and got a batch of twelve. Not twelve thousand. Twelve.
Both Numbers Are Real. Both Are One User.
The headline figures are peak interactivity measurements, and neither vendor claims otherwise — they just don't say it in the press release. Nvidia's 3,400 came from Artificial Analysis, whose headline "Output Speed" metric is defined as "the average number of tokens received per second, after the first token is received." Its concurrency test is a separate run: "we send 10 concurrent requests of our standard 1k input token workload once per day at a random time." The 100K-token workload — the one Nvidia quoted — is run "once per week" and reported as a 14-day median. So the number in the announcement is a single-stream measurement at a context length that is only tested weekly.
Cerebras is explicit that its figure is per user: "in a head-to-head comparison on GPT-OSS-120B, when given identical prompts, the CS-4 delivers more than 4,400 tokens second per user (TPS/user), up to 30 times faster than GPU solutions." Per user, one user.
The two numbers are also not comparable to each other. Nvidia's claimed 4x was measured against Cerebras at 882 tokens per second on the same Gemma model under the same 100K-context conditions — a previous-generation CS-3 result, published before the CS-4 existed. Cerebras' own headline for that model is 1,851 tokens per second, also measured by Artificial Analysis, quoted with no context length attached at all — the same missing qualifier, on the other side of the comparison. Cerebras' 4,400 is a different chip on a different model. Nvidia's own comparison ran 50 back-to-back requests with performance holding steady between 10,000 and 100,000 tokens of input, which is a real and impressive stability result — and still one request in flight at a time.
The 128 GB That Caps Your Concurrency
The ceiling is memory, and on SRAM-only architectures the memory is small. A Groq 3 LPU carries 500 MB of on-chip SRAM. A rack takes up to 256 of them, spread across 2U liquid-cooled trays holding sixteen accelerators each. That is 128 GB of SRAM in the whole rack — against 288 GB on a single Rubin GPU, a 576x difference per chip.
Now spend it. Gemma 4 31B at 8-bit takes about 31 GB, or roughly 64 LPUs, just for weights. That leaves ~97 GB. Every concurrent request then needs its own KV cache — the per-request store of attention keys and values that lets the model avoid recomputing the prompt on every generated token — and at 100K input tokens The Register puts that at about 8 GB per sequence. Ninety-seven divided by eight is twelve. Hence: "the most a single LPX rack with its 256 LPUs can manage on its own before running out of memory is a batch size of 12 at a 100,000 token input length each."
Read that estimate for exactly what it is, because the people who published it are careful about this and most of the coverage repeating them is not. It is a batch size under batch inference with fixed-length inputs, not a headcount of logged-in users. The Register says so directly — "we're assuming batch inference with fixed input sequences, not concurrent users hitting the systems asynchronously, and with different input and output sequences each time" — and adds that for a chatbot on a shorter context window, "a single CS4 or LPX rack could conceivably serve far more users than the 12 noted here." Twelve is the ceiling on simultaneous 100K-token sequences resident in memory. Real traffic arrives asynchronously at mixed lengths and a paged KV allocator fills that space opportunistically, so the number of sessions a rack carries is higher, and the number that matters is still not the one in the press release.
This is not an Nvidia problem. It is an SRAM problem. Run the same arithmetic on a CS-4 — three WSE-3 Turbo wafers at 44 GB of on-processor SRAM apiece, 132 GB total — and The Register lands on the same answer: "we're still looking at a maximum batch size of 12." Two architectures, two vendors, one ceiling, because both bought their per-user speed by putting everything in fast, small, expensive memory.
Nvidia does not dispute the ceiling. Its answer is to not sell the rack by itself. The company's own architecture write-up describes LPX "paired with Vera Rubin NVL72" in disaggregated configurations — prefill on the GPUs and decode on the LPUs, or an attention-FFN split in which "Vera Rubin NVL72 computes attention and holds the KV cache in DRAM" while the LPUs run the feed-forward layers. That last one moves the exact structure that caps the batch off SRAM and into HBM. So The Register's "on its own" is carrying real weight: the twelve describes a standalone rack, and Nvidia is not proposing one. The procurement consequence is not that the constraint evaporates — it is that an LPX quote is a quote for a heterogeneous system, and you need the attached GPU count, the disaggregation mode and where the KV cache lives before any of these numbers mean anything.
The number that moves it is not the hardware. It is your context length. Scale The Register's 8 GB-per-100K-tokens figure down linearly and an 8K-token request costs roughly 0.64 GB, which puts the same rack somewhere near 150 concurrent sequences before overheads. Same silicon, same weights, twelve times the concurrency — because you changed the prompt, not the purchase order. If your agentic traffic really does carry 100K-token contexts, and long-context KV behaviour is where a lot of inference plans quietly break, you are in the batch-of-twelve regime. If your RAG calls are 6-10K, you are not.
Nvidia Publishes the Right Number for GPUs
Here is the part that should decide how you read the LPX announcement: Nvidia already knows how to publish a properly specified inference number, and does it for its GPU line. On InferenceMAX results for Llama 3.3 70B, the company reports that "Blackwell delivers over 10,000 TPS per GPU at 50 TPS per user interactivity" — throughput at a stated per-user speed. That sentence is worth a hundred bare peak figures, because it names both ends of the trade.
InferenceMAX itself exists precisely to sweep that trade: "we set the request rate to infinite and set the max number of concurrent requests, so we capture the inference server behavior when processing a specific number of requests," producing a Pareto curve rather than a point. Its statement of the physics is the cleanest one published: "large batches enable better GPU utilization and higher token throughput, but they split available resources across more requests, slowing down token processing per user."
So the LPX press release is not measuring something dishonest. It is quoting one endpoint of a curve the same company routinely publishes in full elsewhere. Steel-manning the vendors properly: for a coding agent or a voice assistant, the fast end of that curve is genuinely where the value is, and SRAM-heavy designs deliver latency HBM-based systems struggle to match. Cerebras is targeting 10,000 tokens per second per user with CS-5 in 2027, and if your product dies at 40 tokens/sec per user, that matters more than anything on the throughput axis. The failure is not the number. It is a buyer treating a latency number as a capacity number.
What Nutanix Is Actually Buying
Enterprises are spending against these figures right now, which is what makes this a procurement problem and not a benchmarking quibble. Nutanix built a $20 million on-premises AI cluster to cut its spend on Copilot and Claude, with CEO Rajiv Ramaswami expecting to recoup it within a year. His framing of the problem is the one every platform team recognises: "usage exploded and so did costs." The company moved to open-weight models on its own hardware and off per-token billing.
That can be an excellent trade. It is also a trade whose entire payback depends on a denominator — how many concurrent developers that cluster serves at the context lengths a coding agent actually generates. Get the denominator wrong by an order of magnitude and a one-year payback becomes a decade, with the capex already spent.
There is survey evidence consistent with buyers erring in that direction, though it stops well short of proving it. VentureBeat Research surveyed 573 respondents at organizations with 100+ employees in June 2026 and found that "about 83 percent of enterprises that run their own GPUs report utilization of 50% or less" — the piece's own headline says 86% and the body carries a correction to 83%, which is itself a small lesson in reading headline numbers. The same survey found only "a minority 44% rigorously track what their AI compute actually costs and returns."
Apply this article's own standard to that statistic before leaning on it. The utilization figures are self-reported; the survey never discloses how many of the 573 actually run their own GPUs, so the denominator under "83 percent" is unknown; and it offers no cause. Idle silicon is not automatically a sizing error — it can be reserved burst headroom, a training cluster between jobs, or capacity bought ahead of demand on a lead time measured in quarters. Read it as a symptom worth checking in your own fleet, not as a diagnosis of anyone else's.
Half-idle GPUs and a batch-12 rack can be the same mistake pointed in opposite directions: a capacity decision made against a number that did not describe the workload. The installed-versus-energized gap at the hyperscalers is the third face of it. And on-prem economics are moving underneath all of this anyway — server DRAM pricing has been re-pricing the bill of materials while these decisions get made, and Ramaswami cites high hardware prices as a reason he is pushing Arm support, to run on lower-cost hardware.
Rewrite the Acceptance Criterion
Stop accepting tokens per second. Start accepting queries per second at a latency ceiling. That standard already exists and is well specified: MLPerf Inference's Server scenario, where LoadGen "determines the system throughput using multiple test runs," each evaluating a QPS value with Poisson arrivals, and a run only counts if it holds published latency constraints. For Llama2-70B those constraints are 2,000 ms time-to-first-token and 200 ms per output token in the Conversational category, tightening to 450 ms and 40 ms in Interactive. Llama3.1-405B is allowed 6,000 ms / 175 ms, Interactive 4,500 ms / 80 ms.
Note what that structure does. It refuses to let a vendor pick its own operating point. A single-stream figure — MLPerf's Single Stream scenario, where LoadGen "sends an initial query then continually sends the next query as soon as the previous query is processed" — is a legitimate measurement of a different thing, and it is not the one you buy racks against.
Three numbers turn a headline into a capacity plan, and you already have all of them in your gateway logs:
- Peak concurrent in-flight requests, p95, not average.
- Input token length at p50 and p95 — this is the variable that sets your KV cache budget and therefore your batch ceiling.
- The per-output-token latency your product actually needs. Most agentic and RAG workloads are fine at 30-50 tokens/sec per user. If yours is, you are buying on the wrong axis entirely.
What to do
This Week: Pull those three numbers from your AI gateway or serving logs. Do not ask the vendor for them and do not estimate. If your gateway does not emit input-token length per request, that is the week's work — every capacity conversation for the next two years depends on it.
This Month: Re-run any vendor benchmark that is driving a purchase at your own p95 concurrency and your own p95 input length, on the model you will actually deploy. Which serving runtime you use changes the answer materially, so pin that too. Publish the resulting curve internally, not a single number — the same discipline that separates an average agent score from a completion rate.
Before You Sign: Put the acceptance criterion in the contract as sustained concurrent requests at a named context length and a named per-token latency ceiling, measured on your model, with a remedy if it misses. Get the model-scaling story in writing too: The Register notes a 671B model like DeepSeek V3 would need 1,342 accelerators, just over five LPX racks, so a rack price is not a system price. And if you are comparing on-prem to API spend, compare cost per token at your concurrency, not at the vendor's.
The Bottom Line
Every hardware cycle produces a metric that is technically true and commercially useless. Clock speed did it for a decade before anyone made vendors publish performance per watt. Storage did it with sequential read throughput while the workload was random 4K writes. Inference has now found its version: a peak single-stream token rate, published alongside no batch size, no concurrency, and no latency ceiling — while the buyer is holding a $20 million capex request.
The fix is not scepticism about vendors. Nvidia and Cerebras are both selling real engineering, and Nvidia publishes throughput-at-interactivity for its own GPU line. The fix is that the acceptance criterion belongs to you. Silicon diversity is worth having, and betting a rack on one frozen model is a decision you should make deliberately — but none of it helps if the acceptance test came out of the press release.
A benchmark is a measurement of what somebody chose to measure. Choose your own.
Continue Reading
- Groq Runs Nvidia Now. Recount Your Non-Nvidia Capacity.
- vLLM vs TensorRT-LLM vs SGLang: Default to vLLM
- Microsoft's GPUs Sit Unplugged. Buy Delivery, Not Capex.
- Nvidia Is Losing 3 Points to RAM. Re-Price Your Server BOM.
- AMD's Inference Discount Depends on a GPU You Can't Rent
- What RAG Actually Costs: $1,308 a Month at 10M Tokens/Day
- Nvidia Skipped the Prefill. The Math Went With It.