Cloudflare's GLM-5.3 leak: 155 AI answers carried other customers' data because GPU memory was never wiped

Cloudflare has disclosed that its Workers AI service returned 155 responses containing fragments of other customers' prompts and generations from the GLM-5.3 model between August 27 and September 9, 2026. The cause was a mismatch between the model's advertised context window and the size of the GPU cache behind it, combined with a bug in the open-source SGLang inference engine that turned a rejected request into a read of another user's leftover memory. No customer was attacked on purpose, as far as Cloudflare can tell — but the incident shows that the KV cache is now shared tenant memory, and that most AI serving stacks still treat it as a performance detail.

The short version

When a large language model reads a prompt, it builds a working memory of that prompt called the KV cache (key-value cache). On a shared cloud service, that memory lives in a pool of GPU memory divided into fixed-size pages, and the pages are handed from one customer's request to the next. Cloudflare's post is explicit: "Pages are not currently zeroed between requests; the next request that gets a page simply overwrites it."

Cloudflare launched the GLM-5.3 model on Workers AI on August 27, 2026, advertising a context window of 1,048,576 tokens. (GLM-5.3 is the open-weight model whose cyber capabilities Anthropic assessed in October; our coverage.) But the cache on the GPUs that read prompts only had room for about 681,280 tokens. Prompts longer than that were rejected inside the SGLang engine — and the rejection was mishandled. Instead of returning an error, the engine let the request continue with a one-token placeholder, and the GPUs generating the answer then read cache pages that still held other users' data. The customer received an ordinary HTTP 200 response whose text could contain someone else's recent conversation.

A security researcher reported it through Cloudflare's bug bounty program on September 8. Cloudflare took the model offline the next afternoon, fixed the configuration and patched the engine the same day, and published the details on October 26 in coordination with the researcher and the SGLang maintainers.

Timeline

All times UTC, as given by Cloudflare unless noted.

• July 6, 2026 — The same SGLang failure is reported upstream by another user as issue #30233, according to Cloudflare. Cloudflare says it "noticed" this issue "while investigating this incident".
• August 27 — GLM-5.3 launches on Workers AI with a prefill cache pool smaller than its context window. Cloudflare says the exposure was present from launch.
• September 8, 23:11 — Report received via HackerOne from researcher Gadi Evron, with a reproduction script.
• September 9, 08:00 — Cloudflare starts verifying the report.
• September 9, 13:22 — Security incident declared; Product Security reproduces the leak "using two independent accounts within 25 minutes".
• September 9, 14:00 — GLM-5.3 set to private; requests return HTTP 400; status page updated.
• September 9, 14:35 — Root cause identified; a canary prefill node with a larger pool confirmed not susceptible.
• September 9, 14:52 — All prefill nodes moved to the corrected configuration.
• September 9, 19:04 / 19:40 — Deployment rebuilt with the configuration fix and engine patch; model restored to public.
• September 10, 23:30 — Meeting with the SGLang maintainers (RadixArk) on upstream fixes.
• September 11–19 — Additional hardening across all of Cloudflare's production SGLang deployments.
• September 16 – October 12 — Upstream SGLang merges the admission fix, per-transfer checksums and deferred KV release defaults.
• October 2 / October 19 — SGLang v0.5.21 and v0.5.22 released.
• October 26 — Coordinated publication.

What we know

Scope. According to Cloudflare's post, written by Kevin Flansburg and Michelle Chen, "155 responses were affected, all to prompts longer than 681,280 tokens." Affected customers "received responses that contained data from other customers' requests made to the same model shortly before theirs" — prompts and partial generations. Cloudflare says "GLM 5.3 was the only model affected by this bug" and that it "confirmed that no other model hosted on Workers AI was affected", because its other disaggregated deployments had cache pools larger than their context lengths, "so no request could reach the rejection path".

Exploitation. Cloudflare: "We found no evidence of deliberate exploitation." It says the affected requests "are consistent in timing and volume with ordinary long-context use". The post gives the number of affected responses but not the number of affected customers — only "a small number" — and does not describe how responses were matched to customers or whether those whose data leaked were contacted.

The configuration error. SGLang sizes the KV pool "at startup from whatever GPU memory remains after loading the model weights". Cloudflare says it knew before launch that the pool was smaller than the context window, but its long-context tests passed because "we only validated HTTP response codes and engine performance at long contexts", "without validating model output and coherence". The team "assumed that the engine was capable of gracefully handling the difference". Cloudflare calls this "one critical gap in our testing".

The engine bug. Workers AI runs GLM-5.3 with SGLang in a disaggregated configuration: one set of GPUs reads the prompt (prefill) and another generates the answer (decode), with the KV cache shipped between them — in Cloudflare's case using the Mooncake transfer engine (a component with its own security history; see our October 3 report). When prefill rejected an over-length prompt, the rejection path "swapped the prompt for a one-token placeholder so that the request could drain". Admission queues checked whether a request had finished, not whether it had been rejected, so it went through. Prefill sent one page of KV cache; "Decode reserved 8 KV pages for the request" and attended over all of them, including "stale cache pages that still held data from earlier requests". The client got HTTP 200.

The fix. Cloudflare changed GLM-5.3 to match its GLM-5.2 configuration, giving a prefill pool of about 2.04 million tokens — roughly twice the context length. "A prompt that fits the context window now always fits the pool." It also patched SGLang so rejected requests are retired at the admission queue before any cache allocation or transfer, and so the decode side checks that the number of transferred tokens matches its reservation. Further hardening: deferred KV release (pages are held until the transfer is confirmed rather than freed immediately), bounded Mooncake transfer batches "so no single transfer call can outlast the release timeout", and a load-balancer change so "a client disconnect no longer aborts a request mid-transfer". Cloudflare recommends that "all operators update SGLang dependencies to v0.5.22 or later".

Commitments. Cloudflare says it will check pool size against context length "directly, from the running engine" before any disaggregated deployment change ships; build read-time integrity checks that "detect, at read time, that a request is about to attend over a page it does not own"; and apply formal methods, starting with "modeling and proving the safety of an RDMA-based KV cache transfer protocol". A proposal to zero KV pages on release (#46435) is listed as a draft.

Technical analysis

Two ordinary mistakes, one serious outcome. Neither root cause is exotic. An undersized buffer and an error path that does not propagate an error are among the oldest bugs in systems software. What made them a confidentiality failure is the environment: a multi-tenant engine where GPU memory pages pass between customers without being cleared, and where the decode side trusts the prefill side's bookkeeping about how much valid data it sent.

Our assessment: the decisive control failure is the absence of any ownership check at read time. The admission bug let a rejected request through; the size mismatch made the rejection reachable; but the leak happened only because the decode GPU could attend over pages it had reserved but never received data for. In classic operating systems, the equivalent rule — never hand a process memory that still holds another process's data — is enforced by zeroing pages. Here, zeroing is still only a draft proposal, and Cloudflare's own "durable answer" is a read-time integrity check that does not exist yet. Until it does, isolation between tenants rests on every error path in a fast-moving engine being correct.

Why tests missed it. The failure was silent by design: a 200 status, plausible latency, and output text. Testing that checks "did it answer, and how fast" cannot detect "did it answer with the right person's data". Cloudflare's own fix — canary markers, unique strings planted in one account's prompts and searched for in another account's outputs — is the kind of test that would have caught it before launch. It is cheap and should be standard for any shared inference service.

The upstream signal that was missed. The SGLang issue Cloudflare cites, #30233, describes the same mechanism — an aborted prefill reduced to a one-token placeholder, after which decode "starts decoding from incomplete/uninitialized KV cache" — and frames the result as "garbage output". Our assessment: in a shared deployment, "garbage" from another request's memory is not garbage, it is somebody's data. Engine maintainers and operators should treat any report of output built from uninitialised or stale KV state as a potential cross-tenant leak until proven otherwise.

What is new about the AI angle. Prompt-cache isolation has been discussed mostly as a side-channel problem — whether timing differences reveal that another user sent a similar prefix. This incident is more direct: the model was handed another user's cache contents and wrote text conditioned on it. Disaggregated serving, where cache pages cross the network between GPU pools, multiplies the places where an abort, timeout or disconnect can leave a page half-owned. Cloudflare's list of extra hardening — transfer timeouts, mid-transfer disconnects, late RDMA writes landing in reused pages — reads as a map of those places.

Who else is exposed. The SGLang fix is upstream and, in Cloudflare's words, "available for all other operators dependent on SGLang to use". Cloudflare does not say whether any other provider hit the same path, and we found no other disclosure. The precondition — a prefill pool smaller than the context window in disaggregated mode — is a configuration choice, not a default, but it is an easy one to make when a large model barely fits the GPUs. SGLang operators have had a busy autumn: this is separate from the debug-feature code-execution flaw we covered on October 19.

What remains unclear

• Who was affected. Cloudflare gives 155 responses and "a small number of customers", but not how many customers received foreign data, how many customers' data was exposed, or whether either group was notified. The post does not describe direct notification.
• How the 155 were counted. The post does not say which logs were used, how long they are kept, or whether every response in the window could be checked.
• Which SGLang version actually fixes it. Cloudflare's post says both that the patch "was released in SGLang v0.5.20, with follow-ups in v0.5.21 and v0.5.22" and that "In v0.5.21, the SGLang team has now addressed this bug"; it also describes PR #36988 as merged on September 6 but "not yet released" while listing it under v0.5.20. The v0.5.22 release notes on GitHub list related disaggregation fixes but not the PR numbers Cloudflare cites. When we re-checked on November 5, 2026, the post still carried both statements. The safe reading is Cloudflare's own recommendation: v0.5.22 or later.
• Response time. From Cloudflare's timeline, about 15 hours passed between the report (23:11 on September 8) and the model being made private (14:00 on September 9), with verification starting at 08:00. The post does not discuss whether faster takedown was considered.
• The July warning. Cloudflare does not say whether anyone on its team saw issue #30233 before the incident, or whether the SGLang maintainers had triaged it.
• No CVE is mentioned for the SGLang bug, and as of November 5, 2026 we found none assigned, which makes it harder for other operators to track.
• Bounty and researcher method. The post does not say whether a reward was paid or how the researcher found the issue.

Lessons and what to do

For teams running their own inference (SGLang, vLLM, TensorRT-LLM and similar):

• Update SGLang to v0.5.22 or later, as Cloudflare recommends, especially if you use prefill/decode disaggregation.
• Check, from the running engine, that the KV pool can hold your maximum advertised context. If it cannot, lower the advertised context window rather than relying on the engine to reject gracefully.
• Add canary-marker tests across tenants to your release checks: plant unique strings in one account's long prompts and search other accounts' outputs for them. Test the error paths — over-length prompts, client disconnects, transfer timeouts — not just the happy path.
• Treat "incoherent output" bug reports on shared deployments as possible data exposure, and log enough to tell afterwards which request read which pages.

For organisations buying AI through APIs:

• Ask providers how they isolate KV cache and prompt caches between customers, whether pages are cleared or ownership-checked, and how they would notify you if your prompts appeared in someone else's response.
• Keep secrets, credentials and regulated personal data out of very long prompts unless the contract and the provider's isolation story justify it. Long-context requests were exactly the ones affected here.

For leadership:

• AI serving engines are young, change weekly and sit on the confidentiality boundary between customers. Budget for testing what they output, not only whether they stay up.
• Cloudflare's write-up is a useful model of disclosure — root cause, timeline and upstream fixes — but it leaves out the customer-facing questions. Expect your own customers to ask exactly those.

Sources

• Cloudflare Blog — Kevin Flansburg and Michelle Chen, "Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang", October 26, 2026.
• SGLang — issue #30233, "PD disaggregation: aborted prefill requests (input too long) cause decode to generate garbage from incomplete KV cache".
• SGLang — release v0.5.22, October 19, 2026.

Coverage:

• Tech Insider — "Cloudflare GLM-5.3 Workers AI KV cache leak: 155 responses", October 27, 2026.
• AI Weekly — "Cloudflare Says KV Cache Bug Leaked 155 GLM-5.3 Responses".