Cloudflare's GLM-5.3 Leak: How KV Cache Reuse Breaks LLM Tenant Isolation

If you serve prompts longer than your KV cache pool can hold, you are not necessarily getting an error; you may be getting someone else’s data. That is the operational lesson of Cloudflare’s postmortem on a tenant isolation break in its GLM-5.3 Workers AI model, and it converts a question most platform teams treat as a capacity detail, how big is the prefill KV pool?, into a tenant-isolation control that belongs in pre-launch review. Between August 27 and September 9, 2026, Cloudflare reports, customers of @cf/zai-org/glm-5.3 received responses containing data from other customers’ requests: “In total, 155 responses were affected, all to prompts longer than 681,280 tokens.” The company says it “found no evidence of deliberate exploitation” and that the affected traffic is consistent with ordinary long-context use.

Every incident number in this article is Cloudflare-reported from that single postmortem, published October 26, 2026, covering one model and one inference engine over fourteen days. What generalizes is the mechanism and the audit rule it implies; what does not yet generalize is whether vLLM or other providers share the same failure path. Both points matter for the decision in front of you: whether to audit cache-pool sizing before shipping 1M-token document workloads, and what isolation guarantees to demand from a managed endpoint versus a dedicated instance.

What Cloudflare says happened

On September 8, 2026, security researcher Gadi Evron of Knostic reported through Cloudflare’s HackerOne program that very long prompts sent to the GLM-5.3 Workers AI model could return other tenants’ content, per Cloudflare’s disclosure. Cloudflare disabled the model, deployed a corrected configuration and a patched engine, and restored service on September 9. The exposure window began August 27. Cloudflare states no other Workers AI model was affected, because “every other disaggregated deployment on Workers AI has KV pools larger than its context length on both prefill and decode.”

The postmortem attributes the exposure to two overlapping issues. The first was a configuration mismatch, and Cloudflare is unusually direct about it:

We deployed GLM 5.3 with a 1,048,576-token context window, but the KV cache pool on our prefill GPUs only had room for 681,280 tokens. We were aware of this mismatch before launch.

That sentence deserves to be read as a deployment-process finding, not buried as engine trivia. The mismatch was known; the team expected the engine to reject any prompt it could not process, and pre-launch benchmarking returned successful HTTP status codes with clean streaming bodies, so the discrepancy was treated as tolerable. Cloudflare names the testing gap explicitly: it validated response codes and engine performance at long contexts “without validating model output and coherence.”

The second issue was an SGLang bug in how the engine handled over-length requests in disaggregated prefill/decode mode. Cloudflare describes the exposure as “the result of two overlapping issues.” On our reading of that account, neither issue alone would have produced the leak: the undersized pool mattered only because the rejection path was unsafe, and the buggy rejection path was reachable only because the pool was undersized.

The mechanism: a one-token placeholder over stale pages

The KV cache is the per-request attention state a transformer builds during prefill and reads during decode. In a disaggregated deployment, prefill and decode run on different GPU pools, and the cache is transferred between them. Sizing that cache is where the trouble starts, because of a detail in Cloudflare’s account that most operators will not have considered: “the size of this pool is not a configuration value. SGLang sizes it at startup from whatever GPU memory remains after loading the model weights. The model’s advertised context length is a separate configuration value. Nothing forces the two to agree, and it is our job to check that they do.”

So the pool that held 681,280 tokens was an emergent property of leftover memory, while the 1,048,576-token context window was a marketing-facing setting. When a prompt longer than the pool arrived, SGLang’s scheduler rejected it, but the rejection path was the bug. Per the postmortem:

A prompt that did not fit the pool was internally marked as rejected and replaced with a one-token placeholder, but instead of aborting the request and returning a client error, prefill generated the KV cache for the placeholder prompt and transferred it to the decode node.

On a single-node deployment this placeholder design works: the placeholder runs, the batch drains, the client gets an error. In disaggregated mode, the decode node received a “transfer complete” message and “assumed that prefill overwrote the KV cache from previous requests with the full prompt for this request, when in fact it left the majority of the previous state intact.” Decode then generated while attending over stale cache pages still holding earlier requests’ data. Only the first page had been written by the new request; the remaining reserved pages held whatever earlier requests left behind.

This is why the contamination was silent. The HTTP layer looked healthy. The failure only existed in the content of the response, which is precisely what pre-launch testing had not checked.

The remediation, and what the 2x margin buys

Cloudflare’s fix had two halves. First, it matched the GLM 5.3 configuration to its GLM 5.2 configuration, which “freed enough GPU memory that the prefill KV pool now holds about 2.04 million tokens, roughly twice the 1,048,576-token context length.” With the pool larger than the window, a valid prompt always fits and the rejection path is never reached, which, as Cloudflare notes, alone prevents this bug from being exercised. Second, it worked upstream on the engine itself. The SGLang fix for the over-length rejection path landed in v0.5.20 with follow-ups in v0.5.21 and v0.5.22, according to the postmortem, which recommends all operators “update SGLang dependencies to v0.5.22 or later.” Cloudflare also notes it had been flagging related issues with SGLang since v0.5.10 and that a related report sat as issue #30233 from July 6, 2026, evidence that this bug class was visible upstream before it became an incident.

Two cautions on the numbers. The ~2.04M-token pool is Cloudflare’s chosen margin for this deployment, not a demonstrated optimum; the postmortem does not establish that 2x headroom is what your cluster needs, only that pool > context length is the invariant. And the remediation was validated by Cloudflare on Cloudflare’s deployment. The safer reading is: treat “pool exceeds configured context length” as a hard gate, and treat the specific headroom as a judgment call about your own memory budget.

A config-and-audit checklist for SGLang and vLLM operators

Working from the postmortem’s findings, these are the checks that would have caught this class of failure:

1. Read the pool size from the running engine, not from your config. SGLang prints the selected KV pool size in startup logs and exposes it as a metric. Compare it against the advertised context length before any disaggregated deployment change ships. Cloudflare’s own new rule, worth adopting verbatim: “We now check KV pool size against context length directly, from the running engine, before any disaggregated deployment change ships, rather than inferring it from response codes.”
2. Check both sides of a disaggregated deployment. Cloudflare’s isolation held elsewhere because pools exceeded context length on prefill and decode nodes. Auditing only prefill misses half the invariant.
3. Test output content at maximum context, not just status codes. A 200 with a streaming body told Cloudflare nothing about coherence. Long-context acceptance tests should assert something about what the response actually says.
4. Exercise the rejection path deliberately. Send a prompt longer than the pool and confirm the client receives an error, not a successful response. If your engine version predates SGLang v0.5.20, assume this path is unsafe in disaggregated mode until tested.
5. Pin engine currency as a security control. For SGLang, v0.5.22 or later carries the fix plus follow-ups. Version pinning belongs in the same review as pool sizing.
6. Plant canary strings. Because content-level leakage is invisible to HTTP monitoring, embed unique canary strings in long test prompts on multi-tenant deployments and alert if a canary appears in any other session’s output. This is detection, not prevention, but this incident is an existence proof that prevention can fail silently.
7. Scope cache keys per tenant where the engine supports it. This last check addresses the timing side channel covered in the next section rather than the Cloudflare failure mode. Those attacks work because engines compute cache keys over token content alone, with no binding to the requesting principal. The governance study proposes seeding the block-hash chain with a per-principal salt, which it identifies as the mechanism vLLM exposes as a cache_salt request field; by its construction, a salted cache makes one tenant’s probes unable to resolve another tenant’s entries. The cost is losing cross-tenant prefix sharing, so treat scoping as a threat-model decision rather than a default.

Correct sizing is not isolation

Here is the counter-evidence that changes the recommendation: even a perfectly sized shared KV cache leaks information through timing. A governance study of multi-tenant KV caches puts it plainly: “In multi-tenant deployments, this cache is shared across tenants, creating a timing side channel: an adversarial tenant can reconstruct another tenant’s private prompt by probing cache-hit latency patterns.” A cache hit returns faster than a miss; time-to-first-token therefore encodes what is already cached, and what is cached encodes what other tenants asked.

The same paper reports that PROMPTPEEK, presented at NDSS 2025, fingerprinted a victim tenant’s cached prompts by probing TTFT across a known vocabulary and correlating the timing vector against a pre-computed fingerprint, achieving 99–100% attack success rate against SGLang’s radix-tree cache, per the study’s summary of that work. No misconfiguration required. The attack exploits the feature working as designed.

Engine hardening narrows but does not close this. Per the same study, vLLM 0.26.0’s V1 engine dropped block_size=1, which raises the cost of free-text reconstruction, an attacker must now guess sixteen tokens simultaneously at the default block size, but the paper is explicit that this protection “does not extend to template-structured prompts.” If your tenants’ prompts share a predictable template (a system prompt, a RAG wrapper, a tool schema), the structured portion remains probeable.

The consequence for your decision: pool sizing and engine currency protect you against Cloudflare’s failure mode. They do not protect a tenant against a co-tenant who is actively probing. If that threat is in scope, shared-cache endpoints need either a written isolation guarantee from the provider or a dedicated instance where the cache is yours alone.

Why platforms share caches anyway

If shared caches carry this risk, why do they exist? Because the economics of long-context serving push hard toward reuse, and the academic literature quantifies the pressure. KV-Repurpose, a vLLM-based testbed for non-prefix reuse built on vLLM v0.20.0, reports that under a Zipfian RAG-style workload where 20% of requests share a fixed prefix, its approach achieved a 63.9% token-level cache hit rate versus 3.3% for plain prefix caching at the same 30 GB capacity, and that “even with 1 TB of cache capacity, prefix caching reaches only a 17.9% hit rate in this workload.” Exact-prefix matching leaves most reuse on the table at realistic capacities, so the research frontier is aggressive cross-request reuse: KVShare, for instance, explicitly targets multi-tenant KV cache sharing, using a Dual-Stage High Deviation algorithm that recomputes a small portion of the cache during prefill and decode to keep shared state accurate.

There is a correctness cost hiding in that ambition. SAFE-KV frames the constraint: “a cached document state is a context-conditioned causal state, and reusing it outside the context that produced it changes what the model computes.” Reuse is not free even when it is safe from leakage. And in SAFE-KV’s batch-1 L40S study, high-fidelity repair of non-prefix reuse re-encoded so much context that the system’s operating rule reverted to serving every workload by full prefill, a result that undercuts the assumption that more reuse always pays.

These numbers come from different engines, models, and hardware than your deployment, so they do not transfer quantitatively. They do establish the direction of pressure: platforms serving 1M-token contexts have strong incentives to maximize cache reuse and to run pools close to capacity, which raises the cost of a silent sizing error and makes verification your job rather than a default.

SGLang and vLLM: what is known, and what is not

Read the vLLM column carefully: absence of a documented incident is not evidence of absence. The public record simply does not establish whether vLLM, or any other provider’s SGLang deployment, shares Cloudflare’s failure mode. That is an open research question, and the audit checklist above is the bounded verification step that follows from it, not an assurance that any engine is safe.

Managed endpoints versus dedicated instances

This incident breaks a default assumption: that a serverless LLM endpoint isolates requests without you having to ask. Cloudflare’s response was fast and transparent, and the disclosure itself is a model of what to demand, a named external reporter, a bounded window, an affected-response count, and a published remediation. But the exposure window spanned fourteen days and ended only after an outside researcher reported it, and it was caught by content inspection, not by the platform’s own monitoring.

My judgment, given that evidence: if your workload sends prompts above a few hundred thousand tokens through a managed endpoint, ask the provider in writing how the KV cache pool is sized against the advertised context window, whether prefill and decode are disaggregated, and what content-level (not HTTP-level) detection exists for cross-request leakage. A provider that cannot answer those questions has not necessarily done anything wrong; it has not demonstrated the invariant either. If a leaked session would mean rebuilding a customer’s work or exposing privileged documents, the calculus shifts toward a dedicated instance where you own the cache, accepting the throughput you give up as the price of an isolation boundary you can verify yourself. And on any shared endpoint, canary strings and TTFT observation are cheap insurance against both failure modes documented here.

What this evidence cannot settle

Everything specific to the incident comes from one vendor postmortem covering one model and one engine. Cloudflare’s account is detailed and self-critical, but it is still the platform grading its own homework; there is no independent replication of the 155-response figure or the exploitation analysis. The academic results cited, hit rates, attack success rates, recompute costs, come from different engines, models, and hardware and illustrate mechanism and pressure, not your deployment’s numbers. And the question readers most want answered, does my provider, on my engine version, have this problem?, is exactly the one a single postmortem cannot answer. The audit is short, the checks are concrete, and as of this writing the running engine will tell you the truth faster than any disclosure cycle will.

Frequently Asked Questions

What specific SGLang version should operators use to fix the over-length rejection bug?

The SGLang fix for the over-length rejection path landed in v0.5.20 with follow-ups in v0.5.21 and v0.5.22, according to the postmortem, which recommends all operators “update SGLang dependencies to v0.5.22 or later.”

How should operators verify that the KV cache pool is large enough for the context window?

We now check KV pool size against context length directly, from the running engine, before any disaggregated deployment change ships, rather than inferring it from response codes.