In-depth analysis · About 6 min read
Cloudflare explains the GLM-5.3 cache isolation incident and why successful responses were insufficient
The October 26 disclosure separates an August–September exposure from the later engine fixes, configuration checks and unfinished memory-integrity work.
- Published here
- Source published
- Source checked
Cloudflare published an October 26 account of cross-request data exposure in its Workers AI deployment of GLM-5.3. It attributes the incident to a cache-capacity mismatch combined with an SGLang rejection-handling bug. The disclosure describes an incident repaired on September 9, rather than a newly occurring outage or a flaw established in every deployment of the model.
Listen to the news
Read this article aloud or keep listening to the news.
Preparing the reader…
Device voice
Speech is handled by your browser or device. AISubHub does not send the text to an audio API. Voice availability and playback with a locked screen vary by device.
Resuming may repeat a short phrase.
At a glance
- Cloudflare reports 155 affected responses between August 27 and September 9, all for prompts exceeding 681,280 tokens. This is a response count, not a count of customers.
- The deployment advertised 1,048,576 tokens of context while its prefill cache pool held 681,280. The corrected pool holds about 2.04 million tokens; the engine rejection path was also repaired.
- Cloudflare recommends SGLang v0.5.22 or later for disaggregated operators. Detection coverage remains architecture-dependent, and longer-term page-integrity work is not a completed universal guarantee.
Disclosure date and incident scope describe different things
Cloudflare dates the exposure from the model’s Workers AI launch on August 27 through September 9. Researcher Gadi Evron of Knostic reported it on September 8. On September 9, the company declared an incident at 13:22 UTC, restricted the model at 14:00 and restored public service at 19:40 after changes and validation. October 26 is the coordinated publication date.
The company identifies 155 responses containing material from other customers’ recent requests to the same model. Excluding its own and the researcher’s testing, it says the observed timing and volume resembled ordinary long-context use and revealed no evidence of deliberate exploitation. That finding is narrower than proving no misuse occurred. It also says other Workers AI models were unaffected.
The advertised context exceeded the physical cache pool
In this deployment, prefill GPUs processed prompts and transferred the resulting key/value cache through Mooncake over RDMA to decode GPUs. SGLang chose its cache-pool size from memory remaining after loading model weights. The advertised context limit was configured separately. A 1,048,576-token context therefore coexisted with a pool that could hold only 681,280 tokens.
Cloudflare acknowledges knowing about the mismatch before launch, while assuming over-capacity requests would fail safely because long-context tests returned successful responses. The report identifies a testing gap: HTTP success and engine performance were checked without adequate validation of the output. This was a serving configuration and engine interaction; the account supplies no evidence of a backdoor in the model weights.
A rejected request was still treated as a live handoff
The scheduler marked an oversized request for rejection and substituted a one-token placeholder. In the split deployment, however, admission checked whether the request was finished rather than whether it had been rejected. The pending error did not make that finished check true. Prefill and decode consequently accepted a request that should have terminated with an error.
The source diagram illustrates the ownership problem with eight reserved pages: the new request wrote one page, while decode could attend over that page plus seven stale pages. Those page counts are explanatory, not measured incident totals. The client could receive HTTP 200, plausible token accounting and coherent text even though some content came from previous requests. Successful delivery did not establish correct isolation.
The immediate repair had configuration and engine layers
Cloudflare says it changed the GLM-5.3 configuration so the prefill pool held approximately 2.04 million tokens, exceeding the 1,048,576-token context limit. It also repaired rejection handling so a pending rejection terminates at admission, before allocation or cache transfer. The company reports validating both changes before restoring access. Increasing capacity alone would leave a fragile dependency on future memory settings.
The upstream table separates the initial fix in v0.5.20, earlier rejection in v0.5.21 and follow-up hardening through v0.5.22. Cloudflare recommends v0.5.22 or later, or appropriate backports, for disaggregated operators. That recommendation concerns the described serving mode and patch set; it is not a performance comparison or proof that installing a version number makes every deployment safe.
In-flight writes make page lifetime a distributed problem
The report also explains a related failure class: aborting a request cannot recall RDMA writes already in flight. Reusing destination pages too early can let late writes corrupt a new request. Unlike that timing-dependent race, this incident’s rejected handoff followed the wrong path deterministically. Both expose the need to track ownership until every participating operation has finished.
Cloudflare describes deferred release on both prefill and decode, bounded Mooncake transfer batches, changed load-balancer disconnect handling and deployment checks. It recommends measuring actual pools on both roles after changes to weights, quantization or parallelism. Checksum and KV Canary detection do not yet support every architecture; operators must verify coverage rather than assume an enabled setting checks every request.
Longer-term integrity work remains unfinished
A page-tagging prototype was tested on GLM-5.2 in August. Cloudflare reports less than 1% throughput and tail-latency cost in its tests, but says reliability problems led it to disable the prototype shortly afterwards. The number therefore describes a limited experiment, not a currently deployed protection or a general overhead measurement. Clearing pages on release is still listed as a draft proposal.
Transfer checksums check a handoff; page ownership checks would extend validation to later reads. Cloudflare also plans formal modeling of the distributed transfer protocol while acknowledging limits in what proofs can cover. AISubHub analysis: the practical lesson is to validate isolation, rejection and lifecycle invariants separately, and distinguish deployed safeguards from prototypes and future assurance work.
Why it matters
AISubHub analysis: tenant isolation must be an explicit serving-system acceptance criterion. Status codes, token accounting and speed can look healthy while request ownership is wrong; configuration validation and failure-path tests therefore answer different questions from a performance benchmark.
Limits and availability
The incident scope, remediation and absence of evidence of deliberate exploitation are Cloudflare’s account. AISubHub has not independently audited the service or upstream patches. The report does not establish that every GLM deployment was affected, nor that every SGLang configuration is now immune to cache-lifecycle errors.
Sources and evidence
Source published: · Source checked:
AI-assisted editorial articles checked against linked primary sources. Vendor claims are attributed; inclusion is not endorsement.