Cloudflare's GLM-5.3 Leak Exposed Other Users' Data in 155 Responses

If your team runs SGLang with prefill and decode on separate GPUs, or buys long-context inference from a shared serverless endpoint, Cloudflare showed how one customer's prompt can end up in another customer's answer. For two weeks, GLM-5.3 on Workers AI returned data from other customers' requests in 155 responses, all of them to prompts longer than 681,280 tokens. A KV cache pool sized smaller than the advertised context window met an SGLang bug that had been reported upstream in July. Self-hosters should move to SGLang v0.5.22 and check pool size against context length. Buyers should start asking their inference providers how KV memory is isolated between tenants.

Cloudflare published the coordinated disclosure with the SGLang maintainers on October 26.

What Happened on Workers AI Between August 27 and September 9?

GLM-5.3 leaked other users' data from the day it launched on Workers AI until Cloudflare pulled it. Per Cloudflare's timeline, the exposure ran from August 27 to September 9, 2026. Researcher Gadi Evron of Knostic reported it through HackerOne at 23:11 UTC on September 8. Cloudflare's product security team reproduced the leak across two independent accounts within 25 minutes of declaring an incident, made the model private at 14:00 UTC on September 9, and restored it to public at 19:40 the same day with a corrected configuration and a patched engine.

The numbers Cloudflare gives:

• 155 affected responses, after excluding its own testing and the researcher's requests.
• Every one came from a prompt longer than 681,280 tokens.
• "A small number of customers" were affected. The post gives no exact customer count.
• GLM-5.3 was the only model hit. Cloudflare says no other Workers AI model was affected.
• No evidence of deliberate exploitation. The remaining requests matched ordinary long-context use in timing and volume.

The disclosure does not describe how affected customers were notified. If you sent prompts of that size to @cf/zai-org/glm-5.3 in that window, ask your Cloudflare account team directly whether your requests were among the 155, either as the source of the leaked data or as the recipient.

These were not cheap test calls. The model page lists a 1,048,576-token context window at $1.40 per million input tokens, so the affected prompts were the kind teams send when they stuff a whole codebase or contract set into one request.

---

How Did a Rejected Prompt Turn Into Another Customer's Data?

The leak needed two faults at once: a configuration that let prompts exceed the KV pool, and an engine path that treated a rejected request as admitted. A KV cache is the GPU memory where a model stores the intermediate attention state for every token it has already processed, so it does not recompute them for each new token. Inference engines carve that memory into pages and reuse the pages across requests.

The configuration fault. Cloudflare advertised a 1,048,576-token context, but the KV pool on its prefill GPUs held only 681,280 tokens. SGLang sizes that pool at startup from whatever GPU memory is left after the weights load. Cloudflare's post is candid about it: "We were aware of this mismatch before launch." The team expected the engine to reject oversized prompts. Its long-context tests came back HTTP 200 with no error frames, and as the post puts it, "we only validated HTTP response codes and engine performance at long contexts."

The engine fault. Workers AI runs GLM-5.3 with prefill/decode (PD) disaggregation. In SGLang's own description, prefill is the compute-heavy phase that processes the whole input, decode is the memory-heavy phase that generates tokens, and separating them onto different GPUs stops prefill batches from interrupting decode. The KV cache built on prefill GPUs is then shipped to decode GPUs over RDMA, here using Mooncake, the KV-centric serving platform Moonshot AI built for Kimi.

When a prompt overflowed the pool, SGLang marked the request rejected and swapped the prompt for a one-token placeholder. The admission queues checked whether the request had finished(), not whether it had been rejected, so the request went through anyway. Prefill sent one page. Decode had reserved pages for the full prompt and attended over all of them, including stale pages still holding another request's state. The client got HTTP 200 and a response built partly from someone else's context.

None of the usual monitoring would have caught this. Status codes were clean, latency was normal, and the output was fluent text, just conditioned on the wrong customer's data.

Why Did a Bug Reported in July Still Ship in August?

The same failure was filed against SGLang seven weeks before GLM-5.3 launched, and the issue was closed for inactivity during the exposure window. Issue #30233, opened July 6, 2026, describes it exactly: in PD disaggregation mode, an over-length request is aborted on prefill, truncated to a single placeholder token, still enters the queue, and decode "starts from 284936 uninitialized KV cache slots." The reporter reproduced it on SGLang 0.5.11 with GLM-5.2-FP8 on a Mooncake backend, sending 284,937 tokens against a 284,922-token limit, and saw decode return "garbage content unrelated to the input."

GitHub Actions closed the issue as completed on September 5, citing inactivity. The linked fix PR was also closed. Cloudflare's model was leaking at the time.

The steelman for SGLang is fair. The original report reads as a correctness bug, not a security one: "garbage output" on a single-tenant test cluster looks like a quality problem, because there is only one tenant's data in the stale pages. It becomes a confidentiality failure only on a shared deployment where the stale pages belong to someone else. That is the lesson for anyone triaging open-source inference bugs: any report involving uninitialized or reused KV pages is a potential cross-tenant leak once the engine serves more than one customer.

Is This the First Time Shared Caches Have Leaked User Data?

No. The pattern of a cache returning one user's data to another has hit AI services before, and researchers have already shown that shared prompt caches leak information. In March 2023, a bug in the redis-py client library let ChatGPT expose payment-related details of 1.2% of ChatGPT Plus subscribers active between 1 and 10 a.m. PT on March 20. As in Cloudflare's case, a cancellation path corrupted state and returned "information belonging to an unrelated user."

On the inference side, a Stanford team audited prompt caching across LLM APIs and found global cache sharing across users at seven providers, which could leak information about users' prompts through timing. That is a side channel: an attacker infers that someone else sent a similar prefix. The Cloudflare incident is worse in kind, since the leaked state was decoded straight into another customer's output.

What has changed is the architecture. Disaggregated serving moves KV pages between machines on purpose, which adds a transfer protocol, page reservations on two sides and new abort paths. SGLang's v0.5.22 release notes list a PD disaggregation change that delivers "up to 27% more throughput and half the p95 TTFT on long-prompt workloads." Those gains are why providers adopt it, and each one comes with more state that has to be released correctly on every failure path.

---

What Did Cloudflare and SGLang Change?

Cloudflare fixed the configuration, patched the engine so rejected requests never reach KV transfer, and pushed checksum and canary tooling upstream. Per the disclosure:

• Pool size. GLM-5.3 now matches the GLM-5.2 configuration, with a prefill KV pool of about 2.04 million tokens, roughly twice the context length. Cloudflare says of this change: "This alone prevents the bug from being exercised."
• Admission fix. Rejected requests are now retired at the disaggregation admission queue with their original error, before any allocation or transfer. That landed upstream as PRs #36988 (v0.5.20) and #38935 (v0.5.21).
• KV checksums. A --disaggregation-enable-kv-checksum flag (PR #39500, v0.5.21) lets decode verify a checksum of the transferred KV before generating.
• Canaries and metrics. Fixes to SGLang's KV canary and deferred-release metrics, the latter shipped in v0.5.22.
• Hardening. Deferred KV release extended to prefill source pages and all failure types; bounded Mooncake transfer batches; and the load balancer no longer aborts requests mid-transfer when a client disconnects.
• Process. A deployment checklist item that compares KV pool size against context length.

Cloudflare names SGLang v0.5.22 as the minimum for all operators. Its post is not fully consistent on whether v0.5.20 or v0.5.21 first closed the bug, which is one more reason to go straight to v0.5.22 rather than stopping at an intermediate release. Longer term, Cloudflare is prototyping page-level ownership tags that would detect reads of pages a request does not own (under 1% throughput cost in its tests, but disabled in August for reliability reasons) and plans formal verification of its RDMA KV transfer protocol.

What Should You Do About It?

The fix differs depending on whether you run the engine or rent it, and both groups have work this week.

This Week (if you run SGLang):

1. Inventory every SGLang deployment and its version. Anything below v0.5.22 with PD disaggregation enabled goes on the upgrade list first.
2. For each model, read the KV pool size from the startup logs and compare it with the context length you expose. If the pool is smaller, either lower the advertised context or give the pool more memory. Cloudflare's fix ran at roughly 2x context.
3. Send one prompt just over your pool size and check the response code. You want an HTTP 400. A 200 means the reject path is still broken.

This Month (if you run SGLang):

1. Turn on --disaggregation-enable-kv-checksum in a staging environment and measure its overhead before production.
2. Add a canary test to CI: plant a unique marker string in one tenant's long prompt, then search another tenant's outputs for it. Status-code and latency tests did not catch this; an output test would have.
3. Assign someone to watch the SGLang and vLLM trackers for KV, abort and disaggregation bugs, and treat any "garbage output from uninitialized cache" report as a security issue on multi-tenant clusters.

Before Your Next Inference Renewal (if you buy shared endpoints):

1. Add three questions to the vendor security review for Workers AI, Together AI, Fireworks AI or Amazon Bedrock-style endpoints: how KV pages are isolated and zeroed between tenants, whether prefill and decode run on separate nodes, and whether output-level canary tests run at the maximum advertised context.
2. Ask what engine and version serves each model you use, and how fast the provider adopts upstream security fixes.
3. Put a breach-notification clause in the DPA that explicitly covers inference outputs, since the Cloudflare post describes no notification process.
4. If you sent prompts over 681,280 tokens to GLM-5.3 on Workers AI between August 27 and September 9, open a ticket asking whether your data was in any of the 155 responses.

The Bottom Line

The fault sat in how the engine managed GPU memory. Bugs like this show up when a fast-moving open-source engine adds a performance feature faster than its abort paths get hardened, the same way connection-pool bugs in caching clients produced the ChatGPT leak three years ago. Cloudflare's disclosure is unusually complete: timestamps, PR numbers, the admission that the mismatch was known. That detail is what lets everyone else check their own stack.

Long-context requests are where pool sizing, abort handling and transfer protocols get the least testing. Teams that send the largest prompts are the ones running on that path.

Check your KV pool against your context length today, and send your inference provider the three isolation questions before the renewal call.

Continue Reading

• vLLM vs TensorRT-LLM vs SGLang: Default to vLLM
• vLLM 0.30.0 Can Run Code Through Ghostscript Images
• KServe and llm-d Beat Ray Serve for Kubernetes Teams
• Random Eviction Matched the Scorers. Protect the Prompt.
• Together AI and Bedrock beat self-hosted gpt-oss-120b on cost
• LiteLLM's October 21 Compromise Puts Provider Keys at Risk