Only one NVIDIA alternative currently beats it on a published price for silicon you can actually buy: AWS Trainium2. Everything else is either priced above NVIDIA at a hyperscaler, sold at a rate nobody publishes, or attached to a roadmap that has already ended.
That is a narrower answer than the market noise suggests. AMD has a 6-gigawatt agreement with OpenAI and Anthropic runs on over one million Trainium2 chips. None of that is available to you at those terms. What is available is a price list, and the price lists say something specific.
| Accelerator | HBM / chip | Bandwidth | Published price/chip-hr | $ per TB/s-hr | Where you buy it |
|---|---|---|---|---|---|
| AWS Trainium2 | 96 GB | 2.9 TB/s | $2.235 | $0.77 | AWS Capacity Blocks, Ohio |
| Google TPU v6e (3-yr) | 32 GB | 1.64 TB/s | $1.22 | $0.74 | GCP, 3-year commit only |
| Google TPU v6e (on-demand) | 32 GB | 1.64 TB/s | $2.70 | $1.65 | GCP, US regions |
| AMD MI355X (neocloud) | 288 GB | 8.0 TB/s | $2.95 | $0.37 | TensorWave, custom terms |
| AMD MI355X (hyperscaler) | 288 GB | 8.0 TB/s | $8.60 | $1.08 | Oracle OCI on-demand |
| NVIDIA B200 | 180 GB | 8.0 TB/s | $12.355 | $1.54 | AWS Capacity Blocks |
| NVIDIA H100 | 80 GB | 3.35 TB/s | $5.191 | $1.55 | AWS Capacity Blocks |
| Google TPU v7 Ironwood | 192 GB | 7.37 TB/s | not published | — | Negotiated contract |
| Intel Gaudi 3 | 128 GB | 3.7 TB/s | not published | — | IBM Cloud, quote only |
Prices checked 4 September 2026. AWS rates are per-accelerator figures from the EC2 Capacity Blocks pricing page (US East Ohio); Google rates are US-region list; AMD rates are the on-demand and custom rates tracked across providers on getDeploying as of that date.
Why Bandwidth per Dollar Is the Right Denominator
Token generation is memory-bandwidth-bound, not FLOPs-bound. Decoding one token requires streaming the model weights and the KV cache out of HBM; the matrix units spend most of that time waiting. So for a serving fleet — not a training cluster — dollars per terabyte-per-second of HBM bandwidth per hour is the closest single number to dollars per token that you can compute before you buy anything.
Normalise everything to the same workload: a large sparse mixture-of-experts model in the 400B-to-750B total-parameter class, served at 50-130 output tokens per second per user, which is the interactivity band a coding assistant or an interactive chat product needs. That is the workload the table above is priced against.
Two things this denominator does not capture, and you have to hold them separately. Prefill is compute-bound, so a prompt-heavy retrieval workload shifts the advantage back toward raw FLOPs. And a large MoE served with wide expert parallelism becomes interconnect-bound across nodes, where NVIDIA's NVLink rack, Trainium's 2D-torus NeuronLink and Ironwood's 3D mesh are genuinely different machines and no per-chip figure describes them.
The useful property of this metric is that it is an upper bound on your saving. Nobody converts their paper bandwidth advantage in full. The question every option below has to answer is what fraction of it survives contact with the software.
AMD Converts a Quarter of Its Paper Advantage on One Model and All of It on Another
The AMD Instinct MI355X is the strongest alternative silicon in the table and the least predictable in practice. It carries 288 GB of HBM3e at 8 TB/s — more memory than a B200 at the same bandwidth — and on the rates SemiAnalysis assumes in its ownership model ($1.95/hr for a B200, $1.48/hr for an MI355X), that is a 24% paper advantage on bandwidth per dollar.
Here is what it actually delivered. On DeepSeek-R1 at 129 tokens/s/user, SGLang with AMD's MoRI communication library measured the MI355X at $0.169 per million tokens against $0.178 for a B200 running TensorRT-LLM — a 5% win, roughly a fifth of the paper edge. That benchmark is co-published by AMD and the SGLang team, so read it as the vendor's best case rather than a neutral referee. On GLM-5 with multi-token prediction, the same benchmark platform measured $0.22/M against $0.30/M at 18 tok/s/user — a 27% win that fully converts the paper advantage. Note the baselines are not the same: that GLM-5 run puts both chips on SGLang v0.12, while the DeepSeek-R1 comparison above is against B200 on TensorRT-LLM. Against B200 on SGLang, the DeepSeek-R1 gap is 40%, not 5%.
Same silicon, same month, two models, two different answers. The largest variable is whether the kernels for your specific model have landed.
That is not a rhetorical point. vLLM's own engineering write-up on ROCm attention backends reports a 2.7x to 4.4x throughput spread between AMD's own backends on Qwen3-235B — because the model has KV head sizes that HIP paged attention does not support, forcing a fallback to Triton decode kernels. Choosing the wrong backend flag costs you more than the entire NVIDIA-versus-AMD delta.
The harder finding is composability. SemiAnalysis's InferenceX v2 report, published in February 2026, found that AMD holds up when optimisations are tested one at a time, but when you enable the three that production labs actually run together — FP4, disaggregated prefill/decode, and wide expert parallelism — performance "is currently not competitive with Nvidia's." It also found zero MI355X tests in vLLM's CI against multiple B200 tests, and put AMD's distributed inference stack more than six months behind.
Check the date on that report before you use it, because half of it has already moved — which is itself the lesson of this article. Its version-lag figure, an MI355X fork of vLLM 0.10.1 against a 0.15.1 upstream, no longer holds: AMD now documents vLLM 0.27 for MI355X against a 0.28 upstream, on official ROCm images that have been published to the upstream vLLM Docker Hub since January 2026. A five-version gap closed to one in seven months.
The CI gap is the more durable half. AMD's own Q3 2026 vLLM roadmap, filed 31 May 2026, still lists MI355X regression testing and a public no-regression dashboard as goals rather than shipped capability. Thin CI coverage has an operational meaning: upstream regressions on your hardware are discovered by you, in production, not by a maintainer before merge.
AWS Trainium Is the Only Alternative Whose Discount Is Both Published and Real
Trainium2 costs $2.235 per chip-hour on AWS Capacity Blocks against $12.355 for a B200 on the same page, same region, same purchase mechanism. Normalised for bandwidth that is $0.77 versus $1.54 — an exact 2x — and it is the only line in the table where a buyer, a hyperscaler and a benchmark all agree the discount exists.
AWS's own claim is narrower than the paper figure, which is a mark in its favour: 30-40% better price-performance than P5e and P5en instances. Note the honesty gap between 2x on paper and 30-40% realised. That ratio — roughly a third of the paper advantage surviving — is the number to plan against.
Production evidence is the strongest of any alternative here. Anthropic has committed to running its models on Trainium for the next decade, uses over a million Trainium2 chips today, and AWS expects nearly a gigawatt of combined Trainium2 and Trainium3 capacity online by the end of 2026. Project Rainier is in use with nearly half a million Trainium2 chips. AWS also names Databricks, Ricoh and Uber among Trainium users.
The software is further along than its reputation. Neuron SDK 2.32.0, released 17 August 2026, tracks vLLM 0.24.0 through the vLLM-Neuron plugin, adds thirteen new NKI kernels for DeepSeek-V3.2 sparse-MLA context encoding and MXFP8 MoE training, and ships variable-size collectives for ragged per-rank data on Trn2 and Trn3. That is a real inference stack, not a demo.
Two costs are not on the price list. First, the port: a production vLLM stack moving to Neuron means replacing FlashAttention with Neuron's attention kernel and TensorRT-LLM with the Neuron graph compiler, and any custom CUDA kernel has to be rewritten in NKI. Second, and larger, is that when you finish you are on AWS. There is no Trainium anywhere else. You have traded a silicon dependency for a cloud dependency, and only you can price that.
If you want the economics without the port, Amazon Bedrock already serves models on Trainium behind an API. That is the right first move for most buyers.
Google TPU Has the Best Inference Silicon and No Price to Put in a Budget
Ironwood is the strongest inference chip in this comparison and you cannot buy it at a published rate. TPU v7 carries 192 GiB of HBM at 7.37 TB/s per chip, a dual-chiplet design with a die-to-die interface Google describes as six times faster than a single ICI link, scaling to a 9,216-chip pod. Google claims 2x perf-per-watt over Trillium and 42.5 exaflops at pod scale.
It reached general availability on 22 April 2026. Months later there is still no on-demand chip-hour rate. The last generation Google did price, Trillium — 32 GB at 1.638 TB/s — lists at $2.70/chip-hour on-demand, falling to $1.22 on a three-year commitment. At on-demand rates that is $1.65 per TB/s-hour, more expensive than a B200. Only the three-year lock beats NVIDIA, at $0.74.
The absence of an Ironwood price is a procurement fact, not an oversight. Anthropic has access to up to one million TPU chips and well over a gigawatt of capacity coming online in 2026, and has since signed for multiple additional gigawatts with Google and Broadcom starting in 2027. When the marquee customer absorbs that much of the fleet, a retail rate has little to describe. If your budget process needs a number to approve, TPU v7 currently cannot supply one.
The software has genuinely improved. vLLM's tpu-inference plugin unifies JAX and PyTorch under one JAX-to-XLA lowering path, lists v7x, v5e and v6e as recommended, and validates Gemma 4, Llama 3.1/3.3 and Qwen through 3.5-397B. Read the gaps carefully before you plan a migration: hybrid KV cache, multi-host, context parallelism and sequence parallelism are all listed as untested. TensorFlow is not supported on TPU7x at all. Multi-host in particular is the feature that decides whether a 700B MoE fits your topology.
The lower-friction path is the same as with Trainium: serve through Vertex AI and let Google own the port.
Intel Gaudi 3 Is the Loser and You Should Not Start Anything on It
Gaudi 3 is the only option here with no successor. The hardware is respectable — 128 GB of HBM2e at 3.7 TB/s and 1.8 PFlops of FP8, with 24 200-Gbps RDMA ports on-package. The problem is everything around it.
Falcon Shores, its intended successor, was cancelled as a commercial product and kept only as an internal test chip, with Jaguar Shores named as the successor and no date attached. The nearest Intel inference part that has one is Crescent Island, an inference-only Xe3P card with 160 GB of LPDDR5X — not HBM — which begins customer sampling in the second half of 2026. Sampling is not shipping. A part that samples this year is a 2027 purchase at the earliest, and it is a different architecture with a different software stack from Gaudi.
Pricing tells the same story. IBM Cloud, Intel's flagship Gaudi 3 cloud partner, publishes no hourly rate at all — the only number on the product page is "50% off for your first 6 months." A promotional discount is not a price, and a discount is what you offer when the roadmap cannot do the selling.
Then the software — which is the one place Gaudi has genuinely improved, and the honest part of this section. The HabanaAI vLLM fork is end-of-life and deprecated in favour of the vllm-gaudi plugin, which tracks upstream closely: version 0.26.0, built on vLLM 0.26.0, shipped in August 2026. So the version-lag objection that sinks the other alternatives does not apply here. It does not change the conclusion. A current software stack on a product line that terminates still buys you a migration you have to repeat in eighteen months.
If you are already running Gaudi 3 in production, keep serving what works and stop adding models. If you are evaluating it, stop.
What Actually Breaks the Day You Leave CUDA
Six things, and none of them appear in a TCO spreadsheet.
Your serving runtime. TensorRT-LLM has no ROCm, Neuron or TPU equivalent. If your inference path is built on it, you are not porting, you are rewriting. FlashAttention 3 is likewise CUDA-only; AMD substitutes seven different attention backends whose performance varies by more than 4x depending on your model's KV head geometry.
Your custom kernels. Anything using warp-level primitives, cooperative groups or inline PTX needs manual HIP porting on AMD, an NKI rewrite on Trainium, or Pallas on TPU. Automated hipify handles the easy cases and leaves the ones that matter.
Your quantised checkpoints. NVFP4, MXFP4, Neuron's FP8 and TPU's int8 paths are not the same artifact. A quantised model is not portable across vendors; it is a per-vendor build.
Your collectives. NCCL becomes RCCL on AMD, Neuron collectives on Trainium, ICI on TPU — each with different topology assumptions. An expert-parallel layout tuned for an NVLink rack does not transfer to a 2D torus.
Your release cadence. This is the one that compounds. Upstream vLLM ships; NVIDIA support is in that release. As of September 2026 AMD's MI355X path is documented at 0.27, Gaudi's plugin reached 0.26.0 in August and Neuron reached 0.24.0 in the same month — all against a 0.28 upstream. That is closer than this market's reputation suggests, and it is a gap that moves in both directions. Every week of lag is a week your team cannot adopt a speculative-decoding or prefix-caching improvement that your competitor got for free.
Your CI. MI355X regression coverage in vLLM was still a roadmap goal as of May 2026, which means you are the integration test. Budget for that as headcount, not as a one-off migration cost.
The honest counter-argument, and it is a real one: none of this matters if you cannot get NVIDIA hardware at all. Data-centre GPU lead times have run long enough through 2026 that "worse silicon delivered in six weeks" beats "better silicon delivered in nine months." That is a legitimate reason to run an alternative, and it is a different reason from cost. Do not confuse the two in your business case.
Who Should Not Buy Each of These
This is the part a vendor deck will never give you.
Do not stay on NVIDIA if your model set is stable, already validated on Neuron or tpu-inference, and you are single-cloud on AWS or Google anyway. You are paying a 2x bandwidth premium for portability you have decided not to use.
Do not buy Trainium if you serve custom or frequently-changing model architectures, if you are multi-cloud by policy, or if your quarterly volume is not steady enough to amortise a port measured in engineer-weeks. Also skip it if your team's serving expertise is concentrated in TensorRT-LLM.
Do not buy TPU if you need a published price to get budget approval, if nobody on the team can own a JAX or XLA debugging path, or if your topology needs multi-host, context parallelism or sequence parallelism — those are not there yet.
Do not buy AMD if you need FP4 plus disaggregated serving plus wide expert parallelism composed together today, or if procurement will only let you buy from a hyperscaler. At Oracle's $8.60/GPU-hour the MI355X is $1.08 per TB/s-hour — better than a B200, but not by enough to justify a stack migration. The AMD case is made on a neocloud at $2.95, and it dies at hyperscaler rates.
Do not buy Gaudi 3. There is no configuration of buyer for whom starting on a terminated product line is correct.
The One Number That Predicts Regret
Not price. Not TFLOPs. Version lag: the number of days between an upstream vLLM release and your accelerator's backend supporting it.
Almost every failure in this comparison reduces to that number. AMD's 5%-instead-of-24% conversion on DeepSeek-R1 is version lag expressed as kernels. The Ironwood features listed as untested are version lag expressed as a roadmap. Gaudi is the exception that bounds the rule: its plugin caught up to upstream and the silicon roadmap ended anyway, so a low number is necessary and not sufficient.
Measure it before you commit, and measure it on your models, not a vendor's. Pick the three models you actually serve, find the upstream release that added the feature you care about most, and find the date your candidate backend supported it. If the gap is under 30 days, the platform is real. If it is over 90, you are buying a permanent tax on your own roadmap, and the hourly rate will not cover it.
What to Do in the Next 90 Days
This week. Recompute the table above against the rates you are actually quoted, not list. Ask AWS and Google for a Capacity Block or committed-use quote and put your real numbers in the last two columns. Then pull your production model list and check it against the tpu-inference validated models and the Neuron supported architectures. Most evaluations die here, cheaply, which is the point.
This month. Take one model and run it end to end on one alternative, measured in dollars per million tokens at your real interactivity target — not tokens per second at batch one, which flatters every accelerator equally. If you do not want to build that harness, serve the same model through Together AI, Baseten or Bedrock and compare the billed rate against your own fleet. Renting someone else's completed port is a legitimate answer, and often the right one.
Before you sign. Two clauses. A version-lag commitment: the vendor states how far behind upstream vLLM their supported backend runs and what they owe you when it slips. And an exit: what it costs to move your quantised checkpoints and serving config off, with a number attached. Note also that a reserved rate is not a fixed rate — AWS raised Capacity Block base rates roughly 15% in January 2026 across P5, P5e, P5en, P4d, Trn1 and Trn2 in every region at once. Model your alternative against a price that can move.
The Bottom Line
The accelerator market in 2026 looks like the database market in 2010. The challenger is cheaper on paper, genuinely good, and backed by enormous reference customers who negotiated terms you will never see. The incumbent is expensive and boring and every tool works on it the day the tool ships.
The teams that migrated databases successfully were the ones with a small, stable, well-understood workload and a long amortisation horizon. The ones who regretted it were chasing a unit price while their application kept changing underneath them. That maps exactly onto this decision. A stable serving fleet running two or three models you have run for a year is a good Trainium candidate. A team shipping a new model every six weeks is not, at any price.
So: keep the general fleet on NVIDIA. Move a specific, boring, high-volume workload to Trainium if you are already on AWS, and measure it in dollars per million tokens before you move the second one. Treat AMD as a line in your NVIDIA negotiation and a real option on a neocloud, not a migration to fund. Watch TPU and revisit it the day Google publishes an Ironwood price — that publication will be the signal that capacity exists for buyers like you. Leave Gaudi alone.
You are not buying FLOPs. You are buying the distance between an upstream release and your production fleet, and NVIDIA still ships that at zero.
Continue Reading
- vLLM vs TensorRT-LLM vs SGLang: Default to vLLM
- GPU Clouds Compared: Nebius on Price, CoreWeave at 3AM
- 3,400 Tokens/s Was Batch 1. At 100K Context, It's Batch 12.
- Nvidia Buys the Bridge to Trainium. Go Grep Your Imports.
- AWS Orders 1 Million Nvidia GPUs—Then Bets Half on Custom Chips
- Tencent Says 49B Active. Your Node Loads All 770B.
- Equinix Set a Date, Not a Price. Cap Your Commit Term.
- 5% GPU Utilization: The $401B AI Capital Bonfire