Cloudflare has published an incident report that I think is required reading for anyone serving language models: between 27 August and 9 September, the GLM-5.3 model on Workers AI returned to some customers responses containing data from other customers’ requests. There were 155 responses, all to prompts longer than 681,280 tokens.
It was found by researcher Gadi Evron, from Knostic, who reported it through HackerOne with a script to reproduce it. Cloudflare confirmed the bug the next morning, took the model offline and brought it back, fixed, that same day. According to their analysis, there’s no sign anyone exploited it on purpose: the affected requests are consistent with ordinary long-context use. The report comes out alongside one from SGLang, the open source inference engine they use, as a coordinated disclosure.
A bit of context: prefill, decode and the KV cache
When an LLM processes a prompt, it computes key and value tensors for each token, which it later needs to generate the response. That’s the KV cache, and it lives in GPU memory split into fixed-size pages that are assigned to each request and released when it finishes. For performance, pages aren’t zeroed between requests: the next request that gets a page simply overwrites it. That’s standard in inference engines, and it means the engine has to keep very careful track of which page belongs to whom.
Cloudflare serves GLM-5.3 in disaggregated mode: some GPUs do the prefill (process the prompt and build its KV cache) and hand it over RDMA to other GPUs that do the decode (generate the response token by token).
flowchart LR
A["Client · prompt"] --> B["Router"]
B --> C["Prefill · processes the prompt"]
B --> D["Decode · reserves KV pages"]
C -->|"KV cache over RDMA"| D
D --> E["Response"]Two bugs that came together
The first was a misconfiguration. They deployed the model with a 1,048,576-token context window, but on the prefill nodes the KV cache only had room for 681,280. SGLang doesn’t set that size in the configuration: it computes it at startup from the memory left after loading the model weights, and nothing forces it to match the advertised context. They knew this before launch, but in their long-context tests everything returned HTTP 200, so they assumed the engine handled the difference somehow. They admit it themselves: they checked response codes and performance, but not the content of the responses.
The second was an SGLang bug. When a prompt arrived that didn’t fit, the scheduler rejected it and, so the request could drain through the normal path without reserving memory, swapped the prompt for a one-token placeholder. On a single node this worked: the client ended up getting an error. But in disaggregated mode the admission queues checked whether the request had finished, not whether it had been rejected, and a rejected request still looked alive. This is what happened:
| Step | What happened |
|---|---|
| Bootstrap | The router tells decode the real prompt length, and decode reserves, say, 8 pages |
| Prefill | Admits the rejected request and processes only the placeholder token |
| Transfer | Sends a single page and signals that the transfer is complete |
| Decode | Accepts the handoff and generates by reading all 8 reserved pages |
| Response | HTTP 200 with a coherent reply, built partly on the 7 pages that still held data from earlier requests |
The result is particularly treacherous: there was no error. The client got a 200, the prompt token count was right and the reply made sense. The only clue was that it arrived faster than a prefill of that size should take. On top of that, the bug had already been reported to SGLang in July by another, unrelated user.
The fix
They fixed both things, and explain why well: either one alone would have closed this specific case, but not enough to rest easy.
- Configuration: they adjusted the deployment so the prefill KV cache has room for about 2.04 million tokens, twice the context. That way no valid request reaches the rejection path.
- Engine: they patched SGLang so a rejected request is retired, with its error, before any memory is reserved or anything is transferred. Relying only on cache size is fragile, because any change in quantization, parallelism or model version alters it in ways that aren’t obvious.
Then came broader work with the SGLang maintainers to tackle a whole class of similar bugs: pages freed or reused while another part of the system is still using them, whether within a node or between prefill and decode. Among other things, SGLang can now verify a checksum of the transferred cache (--disaggregation-enable-kv-checksum) and delays releasing pages until it confirms there are no pending writes. It’s all in version v0.5.22, and they recommend upgrading to anyone running SGLang in disaggregated mode.
Their recommendations for other operators
The report ends with a practical list, which I’ll summarise:
- Measure the KV cache size and compare it to the context length on every node, after any change to weights, quantization or parallelism. If it doesn’t fit, lower the context. “Do not assume the engine will reject what does not fit.”
- Upgrade SGLang to v0.5.22 or later, as a safety net for the day the previous step is wrong.
- Enable deferred page release on both prefill and decode.
- Enable corruption detection where the model supports it, and check the logs to confirm requests are actually being verified.
- Stop the load balancer from aborting requests mid-transfer because of client disconnects.
Longer term, Cloudflare wants each page to carry a tag that changes when it’s reallocated, so decode can check on every read that the page belongs to that request. It also wants to apply formal verification to the RDMA cache transfer protocol, which ties in with what I wrote a few days ago about formal verification with LLMs.
My takeaway
First, the usual lesson, which is no less useful for being repeated: a 200 doesn’t mean something works. They knew the cache was smaller than the context, tested it and, since they saw no errors, accepted it. It’s a very easy assumption to make with any library: “if it doesn’t fit, it’ll fail”. Here it didn’t fail, and what came out was worse than an error.
Second, this kind of bug is new in how it shows up, but not in its nature. It’s an age-old problem of memory management and of state shared between two machines that disagree, except that now what leaks is other users’ prompts. As models are served with more optimisations (shared caches, disaggregation, reusing memory without clearing it), isolation between customers stops being something you can take for granted.
And third, the report itself. It’s detailed, owns its own mistake plainly, links every fix in SGLang and gives recommendations that are useful to others. That’s how they should all be.
Source: Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang, Cloudflare.
