If your platform team runs Kubernetes and has to pick one way to serve self-hosted LLMs, standardise on KServe's LLMInferenceService and let it run llm-d underneath. KServe and llm-d stack: since llm-d's donation to the CNCF, the project's own description puts it between "high-level control planes (like KServe)" and "low-level inference engines (like vLLM)", and KServe's documentation says LLMInferenceService is built on llm-d. Ray Serve is the real alternative, and it loses the platform-standard slot for most Kubernetes shops: it brings a second scheduler, a second upgrade model and a vendor in the middle of an acquisition. Pick it when your ML engineers already live in Ray. If you serve one model on one node, run plain vLLM and skip all three.
KServe (LLMInferenceService) |
llm-d | Ray Serve LLM | |
|---|---|---|---|
| Layer | Kubernetes control plane: CRD, rollout, autoscaling hooks | Routing and scheduling between gateway and vLLM: KV-cache-aware routing, prefill/decode split | Python-native distributed runtime with its own scheduler (Ray) |
| Runs on | Kubernetes only | Kubernetes only | Kubernetes via KubeRay, VMs, or Anyscale |
| Latest release (checked 2026-10-27) | v0.21.0, Sep 25 | v1.0.0, Oct 20 | Ray 2.59.0, Oct 2 (Serve LLM marked GA) |
| Governance | CNCF Incubating, Nov 2025 | CNCF Sandbox, Mar 2026 | PyTorch Foundation, per Nscale's deal release |
| Main commercial backer | Red Hat (OpenShift AI) | Red Hat, Google, IBM, NVIDIA, CoreWeave | Anyscale, which Nscale agreed to buy |
| Scale to zero | Knative mode yes; KEDA path for LLMs | Supported since v0.5; autoscaler still Tech Preview at Red Hat | min_replicas: 0 |
| Who should skip it | Teams without a Kubernetes platform group | Anyone with one or two replicas per model | Kubernetes platform teams with no Ray users |
| Verdict | Standardise here | Turn on underneath KServe | Second choice; first for Ray-native teams |
What Layer Does Each Project Occupy?
KServe is a control plane, llm-d is a router and scheduler, and Ray Serve is a full runtime, so two of them are designed to run together and the third replaces both. Treating them as three interchangeable "serving frameworks" is how teams end up running llm-d and Ray side by side with nobody owning the request path.
KServe gives you a Kubernetes custom resource for a model. LLMInferenceService is the generative-AI resource; KServe's docs describe it as a CRD that sits alongside the older InferenceService, which the project now positions for predictive models. It handles the deployment, the gateway and HTTPRoute, and multi-node workers: setting spec.worker triggers a LeaderWorkerSet for models too large for one node, and spec.prefill declares a separate prefill-only pool.
llm-d is what makes those features work. Its scheduler (the endpoint picker, or EPP) implements the Kubernetes Gateway API Inference Extension, which lets a standard gateway such as Envoy Gateway or GKE Gateway ask an inference-aware component which pod should take each request. llm-d picks the pod that already holds the request's prefix in its KV cache, splits prefill and decode onto separate replicas, and offloads KV cache across GPU, CPU and storage tiers, per the CNCF announcement. You can install it without KServe, but then you write your own rollout and lifecycle glue.
Ray Serve LLM does all of this inside Ray. Ray's architecture overview says it "takes the performance of a single inference engine (such as vLLM) and extends it" with horizontal scaling, tensor/pipeline/expert parallelism, prefill/decode disaggregation and custom routing, including prefix-aware routing. Ray 2.59.0 marked Serve LLM generally available and added a KV-aware routing guide. On Kubernetes it runs through KubeRay, so Kubernetes schedules the Ray pods and Ray schedules the replicas inside them.
That split decides most of what follows. With KServe and llm-d, Kubernetes owns placement, scaling and rollout; with Ray Serve, Ray owns them and Kubernetes hosts Ray.
When Is Plain vLLM on One Node Enough?
When you serve one model that fits on one node, behind one OpenAI-compatible endpoint, vllm serve is enough, and Ray's own documentation points the same way. Ray describes Serve LLM as an extension of a single engine and says it "excels at highly distributed multi-node inference workloads," citing pipeline parallelism for models too large for one node and cluster-wide parallelism for large mixture-of-experts models (Ray architecture overview). Ray's vLLM compatibility guide adds that you can "switch between vllm serve and Ray Serve LLM with no code changes," which is also an argument for not adopting it before you need it: the engine arguments carry over when you do.
Ray's docs do not print the rule "use vLLM alone below N GPUs." Its small-model tutorial still deploys a 1-2 GPU model through Ray Serve. What the docs make clear is what Ray adds: replicas, autoscaling, multi-model routing and multi-node parallelism. If you need none of those, you are paying for an extra runtime to get the vLLM you already had. The same holds for llm-d: its routing gains come from choosing among replicas, and with one replica there is nothing to choose.
Our vLLM vs TensorRT-LLM vs SGLang comparison covers the engine choice underneath all three; every project here defaults to vLLM.
How Do Autoscaling and Scale-to-Zero Compare for GPUs?
All three can scale a model to zero replicas, but on GPUs scale-to-zero trades cost for a cold start measured in weight-loading time, and only Ray ships it as a plain config flag. Measure that cold start on your largest model before you promise anyone scale-to-zero.
Ray Serve exposes min_replicas. Ray's autoscaling guide says to set it to 0 "if there are long periods of no traffic and some extra tail latency during upscale is acceptable." Setting num_replicas="auto" targets two ongoing requests per replica. When Ray needs more GPUs than the cluster has, the Ray autoscaler requests more nodes, which on Kubernetes means KubeRay asking for more pods.
KServe has two paths. The classic InferenceService scales to zero in Knative (serverless) mode; Red Hat's support note says RawDeployment mode does not scale to zero and costs more at low traffic. For LLMInferenceService, v0.21.0 added direct KEDA scaling, and KServe's overview lists inference-aware autoscaling through the Workload Variant Autoscaler (WVA).
llm-d owns that WVA. The community's v0.5 release introduced scale-to-zero for intermittent workloads, and v0.9.0 moved its autoscaling guide from Prometheus Adapter to KEDA (llm-d releases). WVA scales on GPU saturation signals rather than CPU or request counts: Red Hat's OpenShift AI 3.5 docs say it reads KV-cache utilisation and queue length from vLLM. The same page labels WVA "a Technology Preview feature only." If you buy the commercial version, the autoscaler is not yet supported for production.
Saturation-based scaling is the right signal for LLMs, because a replica with a full KV cache is overloaded long before its request count looks high. Ray's request-count target is simpler and easier to reason about, and it will over- or under-scale on workloads with very uneven prompt lengths.
Which Handles Dozens of Models and Tenants Better?
At dozens of deployments across several teams, KServe plus llm-d wins, because each model is a Kubernetes object that inherits the namespaces, RBAC, quotas and GitOps you already run. Ray makes you choose between many Ray clusters or one shared one.
Define the workload so the comparison is like for like: about 30 models across five teams on one GPU cluster, a mix of 8B-class models with LoRA adapters and two 70B-class models, chat and agent traffic with long shared system prompts, and at least one team that wants a model scaled to zero overnight.
With KServe, that is 30 LLMInferenceService objects in five namespaces. Access, quota and audit follow the namespace. llm-d v1.0.0 adds the part that matters when tenants share a pool: priority- and fairness-aware flow control so latency-sensitive and bursty agent traffic can share a fleet, session affinity that keeps multi-turn conversations near their cached context, router high availability with hot standby, and a DisaggregatedSet that rolls prefill and decode replicas as one unit.
With Ray Serve, model multiplexing packs several models into a shared replica pool, routes by request header and evicts the least recently used model past max_num_models_per_replica, and Serve LLM supports multi-LoRA on a shared base model. That is efficient within one team. Across five teams you either run five RayServices (five Ray head nodes, five upgrade cycles) or one shared cluster where your Kubernetes RBAC stops at the Ray boundary.
Upgrades are where that hurts. KubeRay's default NewCluster upgrade strategy "temporarily creates a new RayCluster" and switches traffic once it is ready, which on GPU nodes means holding spare capacity for a second copy. A NewClusterWithIncrementalUpgrade strategy exists for deployments where duplicating resources is not feasible. Check which one your team will use before you size the cluster.
Who Backs Each Project, and How Mature Is It?
KServe is the most mature, llm-d moves fastest, and Ray carries the most commercial uncertainty right now.
KServe started in 2019 as KFServing under Kubeflow and was accepted as a CNCF Incubating project in November 2025. It ships a minor release roughly every one to two months (v0.20.0 on August 6, v0.21.0 on September 25, per GitHub). It is the serving layer in Red Hat OpenShift AI.
llm-d launched in May 2025 from Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA, and entered the CNCF Sandbox on March 24, 2026, the lowest CNCF maturity tier. It reached v1.0.0 on October 20 with the router configuration API moved to v1. It still churns: v0.10.0 moved the KV-cache code into the router, renamed the autoscaling project and split images between an unsigned dev registry and cosign-signed production images (release notes). Red Hat made it generally available in OpenShift AI 3.0 and sells Red Hat AI Inference "per accelerator," with no published rate (contact sales, checked October 27, 2026). That page also offers early access to llm-d on AWS, Azure and CoreWeave Kubernetes. Our Red Hat AI tool page tracks the product.
Ray is the oldest code of the three and moved to the PyTorch Foundation in 2025. Its commercial steward is Anyscale, and on July 30, 2026 Nscale signed an agreement to acquire Anyscale, expected to close in the second half of 2026. The release says Ray "remains open source and community governed" and that Anyscale keeps its brand and customers. The open-source project has foundation governance; the uncertainty sits with Anyscale's managed platform, as we covered when the deal was announced. Anyscale's pricing page (checked October 27, 2026) bills pay-as-you-go per instance-hour with no listed platform fee, and lists H100, B200 and GB-class GPUs as "Contact Us." See our Anyscale tool page.
How Much Should You Trust the Benchmarks?
Treat every published llm-d and Ray Serve number as a backer's claim measured on the backer's chosen workload, and rerun it on yours before it goes in a business case.
The most cited llm-d result is the September 2025 post KV-cache wins you can see, written by engineers from IBM, Red Hat, Google, Alibaba Cloud and DaoCloud. It reports "57x faster response times," which is P90 time to first token of 0.542 s against 31.083 s for the project's own earlier "approximate" scheduler. Against that same baseline, throughput rose 25% (8,730 vs 6,944.4 output tokens/s); the "double the throughput" line compares against random and load-only routing. The workload was built to reward cache-aware routing: 150 simulated customers, each with a 6,000-token shared context, on 8 vLLM pods across 16 H100s, at roughly 73% of total KV-cache capacity. The CNCF post's figure of ~120k tokens/s comes from a similar 16-H100, Qwen3-32B setup.
Ray's comparable claim is the June 18, 2026 Google Cloud post co-written by Google and Anyscale engineers: up to 5x throughput and 8x better latency than previous Ray Serve configurations, measured on a small model (Gemma 4 E2B) on B200s. That is Ray against older Ray. The post says the new version performs comparably to native vLLM, which tells you the old proxy overhead was real and that Ray 2.56 or later is the floor.
Both workloads were chosen by people who want the project to win. Before you commit, measure four things on your own traffic:
- Prefix share. What fraction of your prompt tokens repeat across requests? llm-d's gain scales with it; at near zero it has little to route on.
- KV-cache pressure. Run at your real concurrency. The 57x figure appears when the cache is close to full and the baseline thrashes.
- The right baseline. Compare against round-robin Kubernetes Service routing and against vLLM's own prefix caching on fewer, larger replicas, not against llm-d's older scheduler.
- Cold start. Time a scale-from-zero on your largest model, from first request to first token.
How to Decide: The Criteria That Predict Regret
The choice that predicts regret is who in your organisation owns the request path from gateway to GPU.
- Your platform team owns Kubernetes and serves models for other teams: KServe with llm-d. Choose this even if the llm-d features are off at first; turning on prefix-aware routing later is a config change, while moving off Ray is a migration.
- Your ML engineers write Python, already run Ray for training or batch data, and want one runtime: Ray Serve LLM. The same Ray cluster runs your preprocessing, your Ray Data jobs and serving, which is the one case where Ray's second scheduler earns its keep.
- You run one or two models with one or two replicas each: plain vLLM behind your existing ingress. Revisit when a second team asks for a model.
Who should not pick KServe: a team without anyone who owns Kubernetes upgrades, Gateway API resources and CRD versions. LLMInferenceService is still an alpha API (Red Hat's OpenShift AI 3.4 docs use v1alpha1), so check which version your install serves before you write manifests.
Who should not pick llm-d: anyone whose traffic shares few prefixes or runs a handful of replicas. You add a router, an EPP and LeaderWorkerSet (v0.11.1 or newer for its prefill/decode guides, per the v1.0.0 notes) for a gain you cannot measure.
Who should not pick Ray Serve: a Kubernetes platform team with no Ray users. You will run a second scheduler you do not know how to debug, plan for duplicated GPU capacity during upgrades, and carry a commercial dependency whose owner changes this half.
What changes the answer: if your models outgrow one node, all three handle multi-node, and that is the case Ray's own docs lead with. If you need cross-cluster or multi-cloud routing, look at the GKE Inference Gateway path Google documents for Ray alongside llm-d's Gateway API integration. GPU placement across teams is a separate decision; see our Run:ai vs Kueue vs Slurm comparison.
What to Do Next
This Week: Inventory every self-hosted model in production: model, replicas, GPUs, owning team and how it is served today. If the answer is fewer than three models with one replica each, stop here and standardise on vllm serve.
This Month: Install KServe v0.21 in a non-production cluster and deploy your busiest model as an LLMInferenceService with llm-d routing on and off. Record P90 TTFT, output tokens/s and KV-cache utilisation for both at your real concurrency, and time a scale-from-zero.
Before Q1 Close: If a team already runs Ray, have them run the same model and traffic on Ray Serve LLM (Ray 2.59 or later) and compare the same four numbers. Pick one platform standard, write it down, and give exceptions an owner and an expiry date.
The Bottom Line
Kubernetes went through this a decade ago with service meshes: the teams that standardised on the platform's own primitives spent their time on traffic policy, and the teams that ran a second control plane spent it on upgrades. LLM serving on Kubernetes looks set to follow the same path, with KServe as the platform resource and llm-d as the GPU-aware routing layer under it. Ray Serve is a capable second platform that most Kubernetes teams do not need to run. Install KServe, turn llm-d's routing on once your own benchmark shows a gain, and keep Ray for the teams already using it.
Continue Reading
- vLLM vs TensorRT-LLM vs SGLang: Default to vLLM
- Run:ai vs Kubernetes vs Slurm: Static Quotas Strand Your GPUs
- Ray Is Open Source. The Control Plane Above It Isn't.
- Red Hat Bets Kubernetes Wins Enterprise AI's Control Plane
- Together AI and Bedrock beat self-hosted gpt-oss-120b on cost
- Random Eviction Matched the Scorers. Protect the Prompt.