AMD's newest accelerator serves the largest open-weight model ever released at roughly 1.5 times Nvidia's tokens per dollar. That result comes from the first serious multi-vendor benchmark on Moonshot's Kimi K3 — and it rests entirely on a rental rate you cannot currently buy.
If you are sizing 2027 self-hosted inference capacity, this is the number that will land on your desk in a vendor deck within a month. It is real. It is also the most fragile figure in the whole comparison, because the perf-per-dollar conclusion is one division problem, and the denominator is an assumption.
What the Benchmark Actually Measured
The benchmark is one of the first measured serving comparisons of AMD and Nvidia silicon on Kimi K3, and AMD wins on cost while losing on speed. On 31 July, Ian Ye at Wafer published throughput numbers for Kimi K3 across three GPU configurations under a post titled "Is memory the moat?" — a 1,024-token input, 400-token output workload run on 8× MI355X, 8× B300, and a two-node 16× B200 deployment.
The headline results:
| Configuration | Single stream (tok/s) | Peak aggregate (tok/s) | Per GPU (tok/s) |
|---|---|---|---|
| 8× MI355X (TP8) | 118 | 952 | 119 |
| 8× B300 (TP8+DCP8) | 172 | 1,568 | 196 |
| 16× B200 (TP16) | 90 | 498 | 31 |
Tensor parallelism — TP8 — means the model's weights are split across eight GPUs inside one server, so every token requires all eight to talk to each other. That matters because the B200 result is not a clean loss: Ye says its numbers "are somewhat deflated by the fact that it pays a cross-node all-reduce on the decode critical path," meaning the 16-GPU configuration spans two physical servers and pays a network round-trip on every token. Kimi K3 needs it. At 2.8 trillion parameters in MXFP4, the weights alone occupy about 1.4TB, and the vLLM team's day-zero support notes put the floor at one 8× B300 node or sixteen B200s. The MI355X and the B300 both carry 288GB of HBM per GPU. The B200 carries 192GB. That capacity gap is the whole reason AMD is in this conversation at all.
Two things about the source belong in the same breath as the numbers. Wafer sells inference optimisation and serverless endpoints for open models, across Nvidia, AMD and other silicon — so a result showing AMD is viable-but-tricky is a result that describes exactly the problem Wafer is paid to solve. And the configurations are not matched: the B300 ran TP8+DCP8 against the MI355X's plain TP8, a difference the table records but the writeup never caveats.
Nvidia wins raw speed, and it is not close. 172 tokens per second single-stream against 118, and 1,568 aggregate against 952. AMD's case is per dollar, never per second.
The $2.50 Assumption Is Doing All the Work
The AMD advantage is not a property of the silicon — it is a property of three rental rates the author picked, and one of them is not purchasable. Ye's perf-per-dollar table assumes $2.50/GPU-hour for the MI355X, $6.00 for the B300 and $4.25 for the B200, sourced from the GPU price aggregator gpus.io.
Check them against the market and the picture inverts.
The B300 assumption is generous to Nvidia. GetDeploying tracks 20 cloud providers across 73 B300 listings; the on-demand rates cluster between Runpod's $6.94 and Sesterce's $8.70, with a median near Nebius's $7.85. The benchmark assumed $6.00 — below the going rate — which makes Nvidia look cheaper than you can actually buy it. Independent trackers agree: Spheron lists B300 on-demand at $9.16/GPU-hour as of July 2026.
The MI355X assumption is the problem. The same tracker finds four providers listing the MI355X, and only one of them publishes an on-demand rate you can actually buy: Oracle Cloud, at $8.60 per GPU-hour. Vultr does list $2.59 — within pennies of the benchmark's assumption — but that capacity is sold out, behind a waitlist, and the trackers cannot even agree what it is: GetDeploying files it as interruptible spot, while GPUPerHour records it as ordinary on-demand billing and states Vultr offers no spot instances for this GPU at all. Hot Aisle and TensorWave are reserved and custom-contract — quote on request, no public number. ComputePrices, updated 2 August, reports a single purchasable data point: one provider, $8.60/hour, 288GB HBM3E at 8,000 GB/s.
So do the comparison at the one vendor that publishes on-demand rates for both chips. Oracle sells the MI355X at $8.60 and the B300 at $15.00.
- MI355X: 119 tok/s per GPU ÷ $8.60 = 13.8 tokens per second per dollar
- B300: 196 tok/s per GPU ÷ $15.00 = 13.1 tokens per second per dollar
A 6% edge. Not 45%. And 6% is the friendliest honest reading, because Oracle's $15.00 is the priciest B300 on-demand listing of the twenty providers tracked. Price AMD's only buyable rate against a B300 at the market median instead and the discount does not shrink — it reverses, to 25.0 tokens per second per dollar against 13.8. Both readings also inherit the benchmark's unmatched topology, which flatters the B300's numerator.
The entire AMD discount lives in the gap between a listing and a rate you can put on a purchase order — and thin supply is the reason that gap exists. Four providers against twenty is not a pricing advantage. It is a single-supplier risk that happens to be quoted cheaply.
AMD's Discount Is a Batch Discount
Where the AMD advantage survives is batch throughput, and it disappears precisely where interactive workloads live. This is the finding that should change what you do, and it comes from a second source with no AMD services to sell.
SemiAnalysis's InferenceX tool compares the same two chips on the previous Kimi generation across three interactivity targets — the per-user token rate your SLA promises:
- At 46 tok/s per user: MI355X delivers 3,375 tok/s/GPU at roughly $0.12/M tokens; B300 delivers 3,230 at roughly $0.19/M. AMD wins on both axes.
- At 72 tok/s per user: B300 pulls ahead on throughput, 2,997 against 2,329. AMD still wins on cost.
- At 97 tok/s per user: B300 delivers 2,667 against 1,679 — 59% more throughput — and is cheaper per token, $0.23 against $0.25.
Two caveats travel with those cost figures. InferenceX does not publish the rental rates or ownership model behind its dollars-per-million — so it rests on an undisclosed denominator, which is the exact weakness this article just spent a section on. And the run is the 1-trillion-parameter K2.6 at 8k input and 1k output, not K3 at 2.8 trillion with a million-token window. Read the crossover as the shape of the trade-off, not as K3's numbers.
The crossover is the story. If you are running overnight document extraction, nightly summarisation, or a batch enrichment pipeline where nobody is watching a cursor blink, AMD is meaningfully cheaper. If you are serving a coding assistant, a voice agent, or a customer-facing chat surface where a user is waiting, the advantage inverts and Nvidia is both faster and cheaper.
The prefill path says the same thing louder. Prefill is the phase where the model reads your prompt before generating anything, and it is what a user experiences as the wait. On an identical 172,000-token prefill, Ye measured 51 seconds time-to-first-token on the MI355X against 23 seconds on the B300 — more than twice as slow. For long-context work, retrieval-heavy agents, or anything that stuffs a codebase into a 1M-token window, that is not a rounding error. It is the product.
The Porting Tax Has an Itemised Bill
The software gap is no longer a theoretical CUDA-moat argument — it is now a list of specific, dated defects with names. Ye's writeup documents three, and they are worth reading as a bill of materials rather than a footnote:
- An undefined symbol. A missing
top_k_renorm_probdefinition in SGLang's ROCm branch crashed the scheduler outright. - A shape mismatch. The AITER MLA prefill kernel required padding the attention heads from 12 to 16 to run at all.
- A fallback cliff. When the optimised kernel does not apply, generic Triton attention delivers 4,000–7,000 tok/s against 13,000 with the tuned path — roughly a 2–3x penalty for landing on the wrong code path.
That last one is structural, not incidental. The vLLM team's February writeup on the ROCm attention backend describes exactly this shape: the HIP paged-attention kernel supports only certain KV head sizes, and unsupported models "fall back to Triton decode kernels, resulting in significantly slower performance." Your model's head dimensions decide whether you get the fast path. Nobody tells you in advance.
Priced in engineering time, switching a model from CUDA to ROCm runs weeks to months for inference-only PyTorch workloads, rising to 6–12 months for distributed training with custom kernels. HIPIFY automates the mechanical translation to under 5% of the codebase; the remaining work is the part that does not automate.
Now steel-man AMD properly, because the trajectory is genuinely good. In MLPerf Inference v6.0 — the same models, same precision constraints, verified by MLCommons — the MI355X hit 92% of B300 offline, 93% server and 104% interactive performance on Llama 2 70B in single-node eight-GPU configurations — though on GPT-OSS-120B it trailed at 91% and 82%. SemiAnalysis, which has spent quarters as AMD's top external bug reporter, reports day-zero support arriving for new frontier models where it used to take months: same-day enablement on MiniMax M3, against the many months it took to reach parity on DeepSeek R1.
The same report names the ceiling: distributed inference "remains model- and topology-specific rather than dependable by default." Single node, AMD is there. Multi-node, it is a project. Note which one a 2.8-trillion-parameter model with a million-token context tends to become.
Price the API Before You Buy Either Chip
Before you compare two GPUs, compare both against not owning GPUs — because at listed rental rates, self-hosting Kimi K3 can cost more than buying it. Ten providers serve Kimi K3, and their prices vary by up to 1.8x; the cheapest tier — Moonshot's own first-party API, Fireworks and Modal, with Together AI also serving the model — charges $3.00 per million input tokens and $15.00 per million output. That is the number your build has to beat.
Working the arithmetic from Ye's throughput figures — 952 tok/s from an MI355X node, 1,568 from a B300 node, at 3,600 seconds an hour:
| Configuration | $/GPU-hr | Cost per 1M output tokens |
|---|---|---|
| 8× MI355X | $2.50 (benchmark assumption) | $5.84 |
| 8× MI355X | $8.60 (Oracle, only public rate) | $20.08 |
| 8× B300 | $6.00 (benchmark assumption) | $8.50 |
| 8× B300 | $7.85 (market median) | $11.13 |
| 8× B300 | $15.00 (Oracle) | $21.26 |
| Cheapest API providers | — | $15.00 |
Two conclusions fall out. At benchmark rates, self-hosting beats the API comfortably on either vendor. At publicly listed on-demand rates, both self-hosted paths cost more per output token than simply buying the tokens.
And every one of those rows assumes your node runs at peak aggregate throughput continuously. It will not. We have covered the 86% of enterprise GPU capacity that sits idle and the utilisation math that turns GPU spend into a capital bonfire. A node at 30% duty cycle triples every figure in that table. The best AMD case, $5.84, becomes $19.45 — above the API price, before you have paid a single engineer.
The licence is the one part of this that is genuinely simple. Kimi K3 ships under a custom licence, not MIT, and its Model-as-a-Service clause triggers only above $20 million in aggregate revenue over twelve consecutive months, with attribution required above 100 million MAU. Purely internal deployments — developers, research, legal review, back-office workflows — sit outside those sections entirely. For most enterprise self-hosting, the licence is not the obstacle. The utilisation is.
Your Move Before 2027 Capacity Locks
This Week:
- Get a real MI355X quote, not a listing. Ask TensorWave, Hot Aisle and Oracle for a written on-demand and reserved rate for the capacity you would actually need. If the number that comes back is $8.60 rather than $2.50, the entire AMD business case changes shape before you have spent an engineering hour on it.
- Write down your interactivity SLA as a number. Tokens per second per user, per workload. That single figure determines which side of the SemiAnalysis crossover you sit on, and therefore which vendor is cheaper for you specifically. Teams that skip this step buy the benchmark's workload instead of their own.
This Month:
- Measure your actual duty cycle on the GPUs you already run. Peak throughput per node is the number in every vendor deck; average utilisation is the number that sets your cost per token. If you are under 40%, the self-host case against a $15 API is much weaker than it looks.
- Price the API path honestly as the baseline. Ten providers means you can shop the rate, and there is no porting cost, no kernel triage and no single-supplier exposure. Build has to beat that, not just beat the other GPU.
- Compute your break-even in tokens, not in dollars. At the benchmark's best case, AMD saves about $2.66 per million output tokens against a B300. Every $1,000 of porting labour therefore needs roughly 376 million output tokens to pay itself back. If your annual volume does not clear that by a comfortable multiple, the answer is not AMD — it is whichever vendor your team already knows.
Before You Sign 2027 Capacity:
- Qualify a second vendor on a workload you can afford to lose. Batch and offline jobs are where AMD's advantage is real and where a Triton fallback costs you latency nobody is measuring. That is a cheap place to build the operational muscle before it is load-bearing — the same logic behind the multi-vendor GPU hedge the hyperscalers are already executing.
- Treat single-supplier risk as a line item with a price. One publicly quoting provider for the MI355X is a supply position, not a market. Ask what happens to your rate at renewal when you have ported four models and have nowhere else to run them — the CUDA lock-in problem in reverse, which is exactly what rationed compute did to enterprises this year.
The Bottom Line
The interesting thing about this benchmark is not that AMD won. It is that AMD won on a division problem where the denominator was an assumption, and the assumption does not survive contact with a rental listing. Cost per token is now the metric the whole industry quotes, and it is the easiest metric in infrastructure to construct backwards from the answer you want — pick the rate, and the silicon follows.
The pattern is old. Every generation of enterprise infrastructure has produced a challenger that was cheaper on paper and expensive in practice: the Linux migration that needed a kernel team, the NoSQL adoption that needed a consistency engineer, the on-prem repatriation that needed a datacentre. The hardware was never the hard part. The staffing was. Kimi K3 beating a frontier closed model on a public coding leaderboard makes self-hosting a genuine option for the first time — and options are worth having, as the open-weight route to sovereignty and the on-prem inference case JPMorgan made both showed. But an option you exercise on the wrong workload is just a cost.
AMD's discount is real, and it is a volume calculation, not a vendor preference. Know your token volume, your duty cycle and your latency SLA before you know your opinion.
Buy the second supplier when the math says so. Never because the benchmark did.
Continue Reading
12GW of AMD GPUs: OpenAI and Meta Break Nvidia's Lock-In Qualcomm Spent $4B to Break Nvidia's Lock on Enterprise AI Cerebras 981 Tok/Sec on Kimi K2.6: GPU Clouds 6.7x Behind JPMorgan Just Ditched Cloud AI. The $117B Reason Why. Free 550B Model: NVIDIA Ends Self-Hosted AI Quality Gap Why 54% Had AI Agent Incidents. 86% of GPUs Run Half-Empty. Your AI Router Is Trading a 10x Discount for a 2.5x One 5% GPU Utilization: The $401B AI Capital Bonfire