In-depth analysis · About 5 min read
Two vLLM KV strategies target different latency bottlenecks
AWS's paired guides contrast push-based hand-off with bidirectional reuse, using separate benchmarks that should not be stacked.
- Source published
- Published here
- Source checked
Two AWS guides published October 19 examine how key-value cache movement affects disaggregated vLLM inference. We cover them together because they describe alternative transfer choices, not two cumulative speed boosts or new releases of the models used for testing.
Listen to the news
Read this article aloud or keep listening to the news.
Preparing the reader…
Device voice
Speech is handled by your browser or device. AISubHub does not send the text to an audio API. Voice availability and playback with a locked screen vary by device.
Resuming may repeat a short phrase.
At a glance
- Push mode reports up to threefold better time to first token and 30% better time per output token for Qwen3-32B on a one-prefill, one-decode P5en deployment with EFA.
- Bidirectional transfer instead reuses decode-side context on later turns, reporting up to 1.6-fold and 3.3-fold TTFT improvements for Qwen3-32B and GLM-5-FP8 under different cache-test conditions.
The shared architecture does not make the tests identical
Prefill processes an input prompt and constructs its KV cache; decode uses that state to generate successive tokens. Separating the engines allows independent scaling, but adds a transfer before generation. Both guides use NIXL and EFA on supported hardware to move device memory without staging through the host. The authors say the features were first developed on Trainium and contributed as accelerator-agnostic capabilities; the reported measurements use P5en GPUs. That is not a claim that every instance has the required direct-memory support. The two optimisations address different waiting periods: preparing a hand-off on one request and recomputing old context on a later turn.
Push moves coordination ahead of completion
Conventional pull waits for prefill, relays block metadata through the router, then allocates and prepares the decode-side read. Push lets decode reserve and register destination blocks while prefill is still running. The producer matches registration and completed computation in either arrival order, then a background thread prepares descriptors and submits NIXL WRITE. Decode waits for every expected completion notification before using the cache. Moving that work off the decode loop can reduce interference with token generation. Both engines use NixlPushConnector for this route. The price is earlier selection of the decode node, whereas pull can defer that decision for load-aware routing after prefill finishes.
Bidirectional reuse needs state that survives a turn
The second design retains decode-side KV for the prompt and generated response, so prefill can retrieve it for the next turn instead of rebuilding the history. A stateful cache-aware router stores block-location metadata by conversation_id, not the KV tensors themselves. The documented default retention is 480 seconds, fixed rather than renewed; later turns fall back to normal prefill after expiry. A default threshold of 64 reusable tokens decides when transfer is worthwhile: smaller amounts are recomputed, while amounts at or above it can be pulled. AWS specifies vLLM 0.21.0 or later, NixlConnector, bidirectional_kv_xfer on both servers and the appropriate router. A new setting alone is not the complete deployment.
Keep the two benchmark baselines separate
Push testing used Qwen3-32B, one prefill and one decode node, P5en with EFA, 128 output tokens, and prefix caching disabled on decode to isolate transfer. At eight requests per second and 16k input, reported TTFT was about 1,744 ms for pull versus 540 ms for push. At heavy load, TPOT was about 10.4 versus 7 ms; short prompts at light load were similar. Bidirectional testing instead disabled prefix caching on prefill to emulate lost local context. Its long-prompt comparisons were 778 versus 460 ms for Qwen at 24k and 2,334 versus 702 ms for GLM at 20k. Those support the reported up-to-1.6 and 3.3 factors within that experiment, not a common ranking across all deployments.
Analysis: choose a cache policy, then validate the service
The push guide explicitly keeps pull as the better fit for bidirectional transfer or deferred decode selection. These are therefore not switches whose published gains should be multiplied together. Our suggested evaluation starts with actual turn spacing, prompt overlap and load, then records TTFT, per-token latency, throughput, cache memory and expired-session behaviour. Disable caching only when reproducing the stated experiment, rather than treating its isolation setting as a universal production recommendation. Request retries, stale metadata and conversation isolation also need operational tests. AISubHub has not deployed either design or measured its bill. The useful outcome is an informed choice between coordination latency, reuse and routing flexibility, with documented conditions rather than a blanket speed claim.
Why it matters
Serving teams can choose whether their main problem is post-prefill coordination or repeated computation across turns. Routing flexibility, network overhead and retained cache memory matter alongside latency.
Limits and availability
These are AWS-reported benchmarks, not AISubHub tests. Hardware, model, traffic and cache settings limit comparison; the gains cannot be multiplied. No consumer quota or fixed inference price follows from the guides.
Sources and evidence
Source published: Oct 19, 2026 · Source checked: Oct 20, 2026
Source published: Oct 19, 2026 · Source checked: Oct 20, 2026
AI-assisted editorial articles checked against linked primary sources. Vendor claims are attributed; inclusion is not endorsement.