If you are sizing inference capacity for coding agents, a GPU utilization number tells you almost nothing about what you will get out of the node. Agent requests carry huge, growing contexts, and once that context falls out of GPU memory the server recomputes it, which keeps the GPU busy without producing anything new. The test that exposes this is a long run on your own traffic pattern that counts only requests that finished inside your latency target. That is goodput, and it belongs in your RFP and your acceptance test.
The prompt for this is the Inference Economics Scorecard that WEKA and NAND Research released at Supercomputing 2026 in Chicago on November 16. It lists nine measures and four testing rules. SemiAnalysis has also published an open trace replayer, AgentX, and a dataset of real Claude Code sessions that let anyone run the same kind of test. The scorecard comes from a storage vendor that sells the fix, so read its benchmark as a vendor result. The method is worth adopting either way.
Why Agent Traffic Breaks GPU Utilization as a Metric
GPU utilization measures whether the chip was busy, and agent workloads keep it busy redoing work. WEKA CEO Liran Zvibel said as much in the scorecard release: "GPU utilization tells you the chip was busy."
The shape of coding-agent traffic is the reason. SemiAnalysis's published dataset card for its v1.0 set of 393 Claude Code sessions totals 21.6 billion input tokens against 106 million output tokens across 98,827 model requests. That averages out to about 219,000 input tokens and 1,077 output tokens per request, roughly 200 tokens read for every token written.
Almost all of that input is context the server has seen before: the same repository files, the same system prompt, the same conversation history, one turn longer. A KV cache is the stored attention state for tokens a model has already processed; keep it and the next turn only computes the new tokens, lose it and the server prefills the whole context again. With prompts averaging about 219,000 tokens, a cache miss means paying for a full prefill to generate a few hundred tokens.
HBM on the GPU is the fastest place to keep that cache and also the smallest. When hundreds of long sessions compete for it, older sessions get evicted, and their next turn arrives to an empty cache. The GPU then reports high utilization while it recomputes context it already processed an hour earlier. Utilization dashboards cannot tell that apart from useful work.
We covered the same failure from the API side, where one timestamp in a prompt can cut caching savings from 80% to 6% and a single pull request can bill 156 million tokens of cache reads. On your own hardware the bill arrives as lost capacity, which is harder to see.
What the Scorecard Actually Measures
The scorecard asks buyers to compare systems on work completed under fixed conditions. Its nine measures, per the release, are tokens per watt, tokens per rack, cost per inference, GPU goodput, time to first token and end-to-end latency, memory efficiency, operational density, agent concurrency, and context retention.
The four testing rules matter more than the list:
- The same workload and service targets across every system tested.
- Measurement of the whole serving stack.
- Runs of at least eight hours.
- Only successful output counted.
Rule three is the one most vendor benchmarks fail. A short run starts with an empty cache, fills it, and ends before eviction starts to bite. An eight-hour run lets sessions age, grow and collide, which is what your production fleet does every working day. Rule four stops a system from looking fast by timing out the hard requests and reporting throughput on the easy ones.
Goodput has a precise meaning in the serving literature. The DistServe paper (OSDI 2024) built its design around it, and its authors' explainer defines goodput as "the number of completed requests per second adhering to the Service Level Objectives (SLOs)." Their worked example sets a time-to-first-token target of 200 ms and a per-token target of 50 ms for at least 90% of requests. Your targets will be different. The scorecard release names goodput as a measure but does not define the thresholds, so you have to write them into the contract yourself.
What WEKA and Nebius Reported on a B300 Node
WEKA and Nebius ran one benchmark under these rules, and the result is large enough to take seriously and specific enough to check. Per Nebius's write-up, one eight-GPU NVIDIA HGX B300 node served DeepSeek-V4-Pro (NVFP4, one-million-token maximum context) on vLLM v0.23 for an eight-hour soak per configuration, replaying a 512-session active pool drawn from more than 5,000 captured agentic sessions. One arm kept the KV cache in HBM only. The other offloaded it to WEKA NeuralMesh Augmented Memory Grid, pooled across local NVMe on an eight-node B300 cluster on Nebius AI Cloud.
The reported numbers, all vendor-run:
| Measure | HBM only | Augmented Memory Grid |
|---|---|---|
| Requests attempted in 8 hours | 23,015 | 55,694 |
| Error rate | 2.32% | 0.64% |
| Cache hit rate | 40% | 93% |
| Median time to first token | 5.68 s | 1.05 s |
| p95 time to first token | 23.12 s | 3.82 s |
| Output tokens per GPU-hour | 46,000 | 169,000 |
The StorageNewsletter republication of WEKA's benchmark post adds the recompute figure: on HBM only, 60% of input tokens were freshly computed, against 6.8% with the offload tier. That is 33,900 fresh tokens per request against 4,400.
Turn the throughput into a unit you can price. At 46,000 output tokens per GPU-hour, a million output tokens takes about 21.7 GPU-hours. At 169,000 it takes about 5.9. Whatever you pay per GPU-hour, that ratio carries straight into cost per inference. The release frames it as the HBM-only node needing about 30 hours to produce what the offloaded node produced in eight.
Where the Benchmark Needs Scrutiny
The strongest case for the result is that it measured the right thing over the right duration and published the error rate. The weaknesses are in what it compared against.
Start with the baseline, which is HBM only. WEKA's benchmark post says no DRAM offload implementation could handle DeepSeek V4 at the time of testing, so the run has no comparison against host-memory offload, the cheapest first step most teams would try. WEKA says a comparison against other approaches will come in a later report. Until then, the 3.7x shows that offload beats no offload and says nothing yet about how WEKA ranks against the alternatives.
The single serving node also had a storage cluster behind it. The GPU count was identical, but the offload arm drew on NVMe pooled across eight B300 nodes, six drives each, per the same post's setup table. In a neocloud cluster those drives often ship in the GPU nodes anyway, which is the best case for the design. On your own hardware, price the drives, the RDMA fabric and the software licence into the comparison.
The two arms ran on different days. The setup table lists July 22 for HBM only and July 26 to 27 for the offload run, with one eight-hour result reported per arm, and a single run per arm cannot show variance.
The traces differ from the public dataset. The benchmark replayed a corpus of more than 5,000 sessions, while the public AgentX dataset has 393. The release says the traces and tooling behind the benchmark are open-sourced through kvlab, so check which trace set a vendor used before you compare its numbers with yours.
Finally, the latency denominator shifts. The 97% against 44% share of requests under a five-second first token is computed over completed requests. Over all attempted requests, StorageNewsletter's table gives 96.3% and 43.3%. The gap is small here, but it is exactly the kind of choice rule four exists to pin down, so specify it in your own test.
The pattern is familiar. Training buyers spent years learning that peak FLOPS meant little next to how many hours a cluster actually trained, which is why training goodput and failover became a purchase criterion. Inference is now going through the same correction, and the quoted batch-1 speeds that collapse at 100K context were an early sign of it.
How to Put Goodput Into Your RFP and Acceptance Test
Write the test before you talk to vendors, and make every one of them run it. The AgentX harness is Apache-2.0, installs with pip, and replays traces against any OpenAI-compatible endpoint, so a neocloud, an on-prem cluster and a managed endpoint can all be held to the same workload. WEKA's kvlab is a second open replay tool with its own capture format.
This Week:
- Pull a week of request logs from your coding-agent gateway and compute your own median input and output tokens per request and your session length distribution. If your average prompt is anywhere near the public dataset's 219,000 tokens, cache retention will dominate your capacity math.
- Write down your service targets: first-token latency, per-token latency and the share of requests that must meet them. Those become the goodput definition in the contract.
- Ask your current inference provider or hardware vendor for their KV-cache hit rate and recompute share on an eight-hour agent replay. Note who can answer.
This Month:
- Run the AgentX harness against your current serving stack for eight hours with the public Claude Code traces, then with a sample of your own sessions if you can capture them. Record requests completed within target, error rate and output tokens per GPU-hour.
- Add a host-memory offload configuration from your serving engine (or a stack like llm-d) to the comparison before you price a dedicated storage tier. The WEKA benchmark skipped this arm, and your test should include it.
- Convert each result to GPU-hours per million successful output tokens and multiply by your actual GPU-hour rate. Put that figure in the business case.
Before Your Next Capacity Reservation:
- Make the acceptance test contractual: same traces, same targets, at least eight hours, failures and timeouts counted against the vendor.
- If a neocloud reservation is on the table, tie the delivery terms to goodput on your workload, and ask for the cache tier's cost as a separate line.
The Bottom Line
The scorecard's metric list will be argued over, and the headline benchmark comes from the company selling the cache tier. The testing rules are sound, and the open harness means you no longer need a vendor to run them. On the published numbers, an HBM-only B300 node computed 60% of its input tokens from scratch against 6.8% with the offload tier, and the difference was mostly context it had already seen. No utilization dashboard would have flagged it. Run the eight-hour replay on your own traces before you sign for more GPUs.
Continue Reading
- Prompt Caching Savings: One Timestamp Turns 80% Into 6%
- One PR Billed 156M Tokens. Cap the Reads, Not the Rate.
- Random Eviction Matched the Scorers. Protect the Prompt.
- Pick DDN EXAScaler for a GPU Cluster You Own
- Together AI and Bedrock beat self-hosted gpt-oss-120b on cost
- Your Nvidia Neocloud Reservation Leaves Build Risk With You
