TensorRT-LLM no longer uses TensorRT. NVIDIA removed the engine-build workflow in version 1.2, and the release notes say it plainly: trtllm-build is gone, and LLM(backend="tensorrt") now raises a ValueError.
That one change invalidates most of the advice you will find on this decision. The standard argument against TensorRT-LLM — that you have to compile a per-model, per-GPU, per-config engine ahead of time and rebuild it every time anything moves — was describing a workflow its own maintainer has since deleted. If your evaluation doc still contains that sentence, you are comparing a product that does not exist.
Here is what to do instead.
Default to vLLM Unless You Can Name the Exception
Run vLLM until you can state, in one sentence, the specific property of your workload that another runtime serves better. That is not a hedge. It is the recommendation, and the burden of proof sits with the alternative.
vLLM is Apache 2.0, hosted under the PyTorch Foundation's neutral governance, runs on the widest hardware surface of the three, and publishes a vulnerability-handling process with CVSS severity tiers and a prenotification group for downstream distributors. None of those are performance properties. All of them are the properties that decide whether you are still running this thing in three years.
Normalise the comparison to one workload, because vendors will not: a 70B-class open-weight model at FP8, on 8×H100, serving mixed agent traffic — roughly 4,000-token prompts with a large shared system-and-tool prefix, 400-token completions, and a p95 time-to-first-token target under 1.5 seconds on fixed capacity. That is the shape most enterprise platform teams are actually holding.
| vLLM | SGLang | TensorRT-LLM | |
|---|---|---|---|
| Pick it when | You want one runtime for everything | Your prompts share long prefixes | You are NVIDIA-only and already pay NVIDIA |
| Licence | Apache 2.0 | Apache 2.0 | Apache 2.0, plus a restricted-licence component |
| Governance | PyTorch Foundation | RadixArk (VC-backed company) | NVIDIA |
| Non-NVIDIA hardware | ROCm, Intel XPU, CPU, Apple | ROCm, Intel, TPU, Ascend | None |
| Published security policy | Yes, with CVSS tiers | No SECURITY.md |
Routes to NVIDIA PSIRT |
| Multi-LoRA under load | Rank-sensitive | Runtime load/unload API | Most stable |
| Structured output overhead | Higher at batch ≥8 | Lowest | Uses the same grammar engine |
| Engine compile step | None | None | Removed in 1.2 |
| Biggest risk | You tune it badly | Single-company steward | Silicon lock-in |
Only two of those nine rows are about speed. That is deliberate, and it is the argument of this piece.
Nvidia Deleted TensorRT From TensorRT-LLM
The product changed identity in two releases, and almost nobody updated their evaluation. TensorRT-LLM 1.0 promoted PyTorch to the default backend and removed the legacy TensorRT documentation. Version 1.2 finished the job: per the release notes, "PyTorch is now the sole execution backend," and TrtLlmArgs, tensorrt_llm._tensorrt_engine.LLM, the trtllm-build / trtllm-refit / trtllm-prune CLIs, the --backend tensorrt flag and the per-model convert_checkpoint.py scripts "have all been removed."
Read that as a buyer, not an engineer. NVIDIA's own architecture overview now describes the project as "architected on PyTorch" with a high-level Python API. The thing that made TensorRT-LLM structurally different from vLLM and SGLang — ahead-of-time graph compilation into a sealed engine — is not a differentiator anymore, because it is not there.
What is left is a very well-optimised PyTorch inference runtime that runs on NVIDIA silicon and nothing else. That is a real product with real strengths. It is just not the product most comparison articles are describing, and the strength it used to be bought for is gone.
The licence deserves one line of attention too. TensorRT-LLM is Apache 2.0, but its licence file bundles third-party components under other terms — including an LTX-2 Community License that restricts commercial use by entities above roughly $10M in annual revenue without a paid licence. That component is for a video model, not text generation, so it is unlikely to touch a 70B chat deployment. Your legal team will still want to know it is in the tree before you vendor the repo.
What "Faster" Means on a Fixed 8xH100 Budget
Every throughput number in every comparison of these three runtimes is stale before it publishes, including the ones in this article. SemiAnalysis built InferenceX, formerly InferenceMAX specifically because of that: it re-runs vLLM, SGLang and TensorRT-LLM across NVIDIA and AMD hardware every time a configuration changes — a new software release, a driver update, a model addition — because a kernel merge or a CUDA point release can reorder the leaderboard inside a week.
So do not buy a runtime on a benchmark screenshot. Go and read the current InferenceX run for your model and your GPU on the day you decide, then decide on the properties that do not move.
The other reason to distrust a headline number is that your own configuration dominates it. The team at AI21 published a tuning writeup that is worth more than any vendor chart: on a 32B model on H100s, moving --max-num-seqs from 64 to 1,792, --max-num-batched-tokens from the 2,000 default to 49,000, and tensor parallelism from 1 to 4 delivered roughly 2× the throughput and 2× lower latency from the same four GPUs. A 2× swing from flags on one runtime is larger than the gap most benchmarks report between runtimes.
If your evaluation shows a 15% difference between two engines and you have not tuned either, you have measured your defaults.
Where SGLang Actually Wins: Shared Prefixes
SGLang's advantage is structural, not incremental: RadixAttention keeps a global radix tree of KV cache across requests, so a shared prefix is computed once for everyone rather than once per session. For the workload defined above — agent traffic where every request carries the same multi-thousand-token system prompt and tool schema — that is the single highest-leverage property on the list.
It is also the most deployed. SGLang's repository lists xAI, NVIDIA, AMD, LinkedIn, Cursor, Oracle Cloud, Google Cloud, Microsoft Azure and AWS among its users and claims it powers over 400,000 GPUs. Coverage of its commercial spin-out puts it across hundreds of thousands of GPUs, generating trillions of tokens a day, for customers including Google, Microsoft and xAI — a launch announcement, so read the numbers as the company's own, but the named adopters are checkable. That is not a research project.
The second SGLang advantage is quieter and matters more for agents than most teams realise. All three runtimes use the same grammar engine family for constrained decoding, so JSON-schema enforcement is table stakes — but the integration differs. An independent benchmark from SqueezeBits on H100s with Qwen3-8B and Qwen3-32B found vLLM showed "significant performance drop with guided decoding compared to its baseline, especially at a batch size of 8 or greater," while SGLang achieved structured output "with minimal performance loss", because it overlaps CPU-bound grammar work with GPU inference at the scheduler level. If every one of your requests is a tool call, that overhead is not an edge case — it is the workload.
SGLang also ships fast. Its releases page shows v0.5.18 landing 710 pull requests from 212 contributors, roughly a fortnight after the previous version. Read that as velocity if you have a platform team that tracks upstream, and as churn if you do not.
Multi-LoRA Is the One Test TensorRT-LLM Wins
If you serve dozens of fine-tuned adapters off a shared base model, TensorRT-LLM is measurably the most stable option, and this is the strongest argument against my own recommendation. SqueezeBits benchmarked 16 LoRA modules on Llama-3.1-8B on an A100 80GB at batch size 256, and the two frameworks fail differently. Applied to all linear layers, vLLM degraded 23.9% to 47.0% against the base model and TensorRT-LLM 40.0% to 47.7%. Restrict the adapter to query-value layers only and TensorRT-LLM improves sharply, to 14.1% to 18.6%, while vLLM barely moves, at 23.8% to 46.1%. The writeup's own framing is that vLLM is the more sensitive to LoRA rank and TensorRT-LLM to which layers you target — TensorRT-LLM stayed flat as rank rose from 8 to 64 where vLLM's throughput fell. Under memory pressure with 64 adapters, TensorRT-LLM's guaranteed-no-evict scheduling held throughput while vLLM hit preemption.
Take that seriously if adapters are your architecture. Take the operational side seriously too: the same writeup found TensorRT-LLM less intuitive to operate, and vLLM better suited to frequent adapter updates.
The runtime ergonomics differ in a way the benchmark does not capture. SGLang exposes /load_lora_adapter and /unload_lora_adapter HTTP endpoints for adding adapters after startup, with max_loras_per_batch defaulting to 8. vLLM supports dynamic loading too, but its own documentation warns that it "comes with security risks" and "should not be used in production unless it is an isolated, fully trusted environment" — a LoRA path is arbitrary weights loaded from disk, and that is a real supply-chain surface, not a theoretical one.
If your adapters change weekly and are published by internal teams, vLLM's flexibility is worth its overhead. If you have fifty stable adapters and a latency SLO, TensorRT-LLM earns its keep.
Hardware Portability Is the Only Irreversible Choice
Everything else in this decision is reversible in a sprint; the silicon you standardise on is not. vLLM's installation matrix covers NVIDIA CUDA, AMD ROCm, Intel XPU, x86 and ARM CPUs, IBM Z and Apple silicon, plus third-party plugin backends. SGLang lists NVIDIA, AMD, Intel, Google TPU and Ascend NPU across its recent releases. TensorRT-LLM runs on NVIDIA GPUs. That is the whole list, by design, and it is not going to change.
Quantization is where that constraint bites first, and the honest version is messier than either vendor's marketing. NVFP4 is Blackwell-specific; MXFP4 is the 4-bit format both vendors support natively, which makes it the only real choice for a mixed fleet. AMD has built an NVFP4 path into vLLM for MI355, but read its own caveat rather than the headline: the pipeline dequantizes NVFP4 weights to BF16 at GEMM time, and AMD states plainly that the added compute overhead makes it better suited as an "accuracy reference for quantized model validation than a production serving deployment". Portability is not the same as parity. vLLM's own quantization compatibility matrix spans Volta through Hopper plus AMD and Intel GPUs and x86 and Arm CPUs, but the fast path still depends on picking a format both your vendors run natively. Whether AMD makes commercial sense at all is a separate question — one we have looked at through the cost-per-token math on MI355X versus B300 — but choosing TensorRT-LLM forecloses the option before you get to ask it.
The counterweight, stated fairly: TensorRT-LLM is the reference implementation NVIDIA optimises new silicon against, and it is one of the engines NVIDIA packages inside its own NIM microservices — though those auto-select between TensorRT-LLM, vLLM and SGLang per model, which is itself a data point. On day-zero support for a new Blackwell or Rubin part, it will lead.
One structural relief on all of this. NVIDIA's own orchestration layer, Dynamo, treats all three as pluggable backends beneath a common disaggregated-serving and KV-routing layer — its repository puts it directly: "it doesn't replace SGLang, TensorRT-LLM, or vLLM, it turns them into a coordinated multi-node inference system." When the vendor with the strongest incentive to lock you into its own runtime instead builds a layer that makes all three interchangeable, that tells you something about how durable the differences are. The runtime is a less permanent commitment than it feels like — provided you keep your model weights, your eval harness and your routing layer outside it. The lock-in that hurts is upstream in the silicon and downstream in the control plane, which is the same lesson that landed when Anyscale's commercial layer above open-source Ray changed hands, and again when d-Matrix bought Wallaroo and "any hardware" turned out to need writing down.
The Governance Question Nobody Puts in the Table
Ask each project who is legally responsible for it and how it handles a vulnerability, because the answers are strikingly different and none of them appear in a throughput chart.
vLLM sits in the PyTorch Foundation under neutral IP governance. Its security policy defines four CVSS-banded severity tiers, develops critical and high-severity fixes in private forks, and runs a prenotification group that downstream distributors can join. That is what a mature process looks like, and it is why Red Hat can sell a supported vLLM and llm-d distribution priced per accelerator — a route worth knowing about if your procurement process needs a throat to choke. Its validated Kubernetes stack bundles "vLLM, KServe, llm-d, a dedicated Istio instance (managed by the Sail Operator), cert-manager, LWS, and the Gateway API", which is a useful reference architecture even if you never buy the support.
TensorRT-LLM routes vulnerabilities to NVIDIA PSIRT with a formal submission form, PGP key and published bulletins. Corporate, centralised, and entirely adequate.
SGLang's security policy page says: "This project has not set up a SECURITY.md file yet." You can still report a vulnerability through GitHub, but there is no documented triage process, no severity taxonomy, no disclosure timeline. For a bank, an insurer or a public agency, that is a procurement finding, not a nitpick — and it is fixable upstream in an afternoon, which is exactly why its absence is informative.
The stewardship question is newer. In January 2026 TechCrunch reported SGLang spinning out as RadixArk at a roughly $400M valuation, and in May the company launched with $100M in seed funding at a $400M post-money valuation, led by Accel and Spark Capital, with NVIDIA's NVentures and AMD participating. It was founded by SGLang's own creators, Ying Sheng and Banghua Zhu.
Steel-man that properly: a funded company with paid maintainers is often better for a critical dependency than volunteer stewardship, and the code stays Apache 2.0 whatever happens. But you now hold a dependency whose roadmap answers to a cap table, and the open-core pattern is well documented — the engine stays open while the operational layer above it becomes the product. That is not a prediction. It is a risk to price, the same way you would price it for any other single-vendor dependency in the stack.
What This Costs When You Don't Build It
All three runtimes are free; the decision is never about licence cost, it is about who absorbs the tuning and the on-call. Rent the hardware and the arithmetic is public: Lambda lists H100 SXM on demand at $3.99 to $4.29 per GPU-hour and B200 SXM6 at $6.69 to $6.99, checked 24 August 2026.
Against that, price the two escape routes honestly.
Managed serving takes the runtime problem away entirely. Baseten publishes dedicated-deployment pricing at $0.10833 per minute for an H100 — $6.50 an hour — and $0.16633 per minute for a B200, $9.98 an hour, as of 24 August 2026. Call it a ~$2.30/GPU-hour premium over raw Lambda H100 capacity. On eight GPUs run continuously that is roughly $13,000 a month for someone else to own the OOM at 3am. If you do not have a platform team who wants that pager, it is cheap. Fireworks AI, Together AI and Modal sit in the same category.
NVIDIA's packaged path is the other one, and its pricing is oddly hard to pin down. NVIDIA does not publish a list price on either its AI Enterprise product page or its FAQ — both route you to sales. The AWS Marketplace listing does publish rates, and they resolve to about $1.00 per GPU-hour: $1.00/hour on a single-GPU g5.xlarge, $8.00/hour on an eight-GPU p5.48xlarge, metered only while the instance runs. That is roughly a 25% surcharge on the H100 rental itself, for support and the packaged microservices.
That $1/GPU-hour figure is the one number that makes TensorRT-LLM's case coherent. If you are already paying it for support and vGPU and orchestration, the marginal cost of using NVIDIA's runtime is zero, and the integration is done for you. If you are not paying it, you are choosing single-vendor silicon lock-in and getting no support in exchange.
Whatever you choose, keep the accounting outside the runtime. Token metering, per-team chargeback and provider failover belong in a gateway, for the reasons we set out in the case for self-hosting your LLM gateway first — and cache economics now dominate routing decisions in a way that broke the multi-model router thesis outright.
Who Should Not Pick Each of These
This is the section a vendor page will never write, so it is the one worth reading twice.
Do not pick vLLM if your entire workload is dozens of LoRA adapters against one base model with a hard latency SLO — the rank sensitivity in the SqueezeBits numbers is real. Do not pick it if guided decoding is on every single request at high concurrency and you have measured the batch-size-8 cliff on your own traffic. And do not pick it expecting it to be fast out of the box; the AI21 result shows the defaults leave a 2× swing on the table, and a runtime that rewards tuning punishes teams who will not do it.
Do not pick SGLang if your security or procurement process requires a documented vulnerability-disclosure policy from every open-source dependency, because today there is not one. Do not pick it if you have no platform engineer tracking upstream — a two-week release train with 700 merged PRs is a gift to a team that reads changelogs and a liability to one that does not. And do not pick it for a workload with no prefix sharing at all: batch document processing over unique inputs takes almost none of the RadixAttention benefit while taking all of the governance risk.
Do not pick TensorRT-LLM if there is any chance you will run AMD, Intel, TPU or a custom accelerator in the next three years — that door closes the day you standardise. Do not pick it if you are not already buying NVIDIA AI Enterprise, because you are paying the lock-in premium without collecting the support. And do not pick it because a 2025 benchmark said it was fastest; the workflow that produced that gap was deleted in 1.2, and the current stable release is 1.2 while the project ships 1.3 release candidates.
The loser, named: for the enterprise platform team in the buyer profile above, TensorRT-LLM is the weakest of the three. Not because it is slow — on NVIDIA hardware it is excellent and it is what NVIDIA optimises new silicon against. It loses because the single property that justified its operational cost was removed by its own maintainer, and what remains is the only option in the set that is single-vendor by construction — at the precise moment a second silicon vendor became a real option for anyone willing to standardise on MXFP4. If you cannot say "we are NVIDIA-only forever and we already pay for AI Enterprise," it is not your default.
Three Checks Before You Commit the Runtime
This Week: Pull the current InferenceX run for your exact model and GPU. Then run your own load test at your real prompt shape — measure prefix overlap explicitly, because that single number decides whether SGLang's core advantage applies to you or not.
This Month: Tune before you compare. Set --max-num-seqs, --max-num-batched-tokens and tensor-parallel size deliberately on every candidate, then re-measure. An untuned bake-off measures defaults, not engines. Separately, get the security policy question answered in writing by whoever owns third-party risk — that is a two-line email that can save a six-week rollback.
Before Your Next Capacity Commit: Write down what happens if you have to move. Keep weights, eval harness, routing and metering outside the runtime, and confirm your orchestration layer treats the engine as replaceable — Dynamo does, and so does the vLLM plus llm-d plus KServe stack. Then check your Q3 roadmap dependencies against the runtime's own; vLLM is deprecating Model Runner V1 and moving its Rust frontend to production, and you would rather learn that from a roadmap than from a broken upgrade.
The Bottom Line
This decision looks like a performance question and is not one. The performance gaps between vLLM, SGLang and TensorRT-LLM are real, measurable, and smaller than the gap between a tuned and an untuned deployment of any one of them — and they get reordered on every release by people with more GPUs than you have. Meanwhile the things that are genuinely hard to reverse — which silicon you can buy next year, who is legally responsible for a CVE, whose cap table your dependency answers to — appear in none of the benchmarks anyone is arguing about.
We have seen this shape before. The database wars were settled by operational maturity and portability, not by TPC-C scores, and every team that picked the fastest engine of 2004 spent 2008 migrating off it. Inference runtimes are at the same point: the engines are converging fast enough that NVIDIA's own runtime now ships code borrowed from both of its competitors under Apache 2.0, and its orchestration layer treats all three as interchangeable.
Pick the one you can still leave.
Continue Reading
- Self-Host the Vector DB for Residency. Not for the Bill.
- AMD's Inference Discount Depends on a GPU You Can't Rent
- Best LLM Gateways for Cost Control: Self-Host First
- Ray Is Open Source. The Control Plane Above It Isn't.
- Nvidia Skipped the Prefill. The Math Went With It.
- Groq Runs Nvidia Now. Recount Your Non-Nvidia Capacity.
- DeepSeek Will Raise Prices. Your Ceiling Is Already 4x.
- 7 OpenAI Alternatives. Only 3 Clear a Sovereignty Rule.