[Blog]: P2P KV Cache Sharing Under Agentic Serving at Scale

[nilig]

Contact Details

@niliguy, @liranschour

Proposed Title

P2P KV Cache Sharing Under Agentic Serving at Scale

Short Synopsis

Follow-up to Pull, Don't Recompute, which priced the transfer and showed three payoff cases. This post measures P2P against the agentic traffic shape at fleet scale: long prefixes re-read almost verbatim across turns, sub-agent fan-out arriving with no advance signal, and sessions that pause and then burst. It covers the four cases the first post named as unmeasured - hot-prefix skew, scale-out warmup of a cold replica, restart and preemption recovery under live multi-turn sessions, and prefetch ahead of arrival - plus a cross-vendor crossover measurement so the minCachedTokenDelta guidance is not tied to one accelerator.

Why This Post?

The published post establishes the economics and three payoff cases, but explicitly leaves fleet dynamics for later: those are the cases operators hit in production, where locality breaks arrive continuously rather than once. Agentic serving is also where the mechanism should pay the most, since nearly every turn re-reads a prefix the cluster already computed.

The published numbers are single-vendor: every benchmark ran on H200 with NIXL/UCX over rdma/ib, and the guide's prerequisites name CUDA GPUs. Whether the pull-versus-recompute crossover, and therefore the recommended minCachedTokenDelta, transfers to other accelerators is unmeasured, so operators on those stacks have no calibration starting point.

Target Audience

Operators running agentic or coding-assistant workloads on multi-replica llm-d deployments; contributors working on KV cache management, P/D disaggregation, and the P2P connector.

Who Will Write It?

I'm open to writing it or finding a co-author

Related Links

• https://llm-d.ai/blog/p2p-kv-cache-sharing-llm-d
• https://github.com/llm-d/llm-d/tree/main/guides/p2p-kv-cache-sharing
• https://llm-d.ai/blog/serving-glm-5-2-agentic-workloads-on-llm-d
• [Blog]: Peer-to-Peer KV Cache Sharing in llm-d #425

Additional Context

Partners with agentic production traffic or non-NVIDIA hardware are welcome to co-author. @PrateekKumar1709 (AMD) offered to collaborate in #425; the contained first piece of work is the Step 0 ladder from the benchmarking guide - recompute versus pull on a single pair with no router in the path - which establishes both whether the offload/NIXL path works on that stack and where the crossover sits.

[PrateekKumar1709]

Thank you @nilig, happy to contribute. Following on from what I offered in #425, we would be glad to co-author. If we are able to run this on AMD Instinct hardware, that could also give the post the non-NVIDIA coverage this is looking for.

@shikamd123 from AMD is also interested in collaborating on this, so please keep us both in the loop. We are open to taking a share of the writing, and to working alongside whoever else picks it up.