State-of-the-Art KV Transfer for Disaggregated LLM Serving in vLLM#
PyTorch Conference 2026 digest Program Core pick 20–21 October 2026
Nicolò Lucchesi (Mistral AI, vLLM maintainer), Sunita Nadampalli (Amazon), Zhanqiu Hu (Red Hat)
- Why: P/D disaggregation turns KV-cache transfer into a first-class network workload — bidirectional, latency-sensitive, stateful — that inference fabrics must now be designed for. This is the talk defining that transfer layer's requirements.
- What: Developments in vLLM's disaggregated serving stack: hybrid-model transfer with heterogeneous TP (Nemotron, Qwen3.5, DeepSeek-V4 with interleaved attention, KV compression, sliding windows), bidirectional KV transfer for agentic multi-turn sessions, a KV Push connector that reduces TTFT, and KV cache leases for reliability under failures and traffic spikes. On Nemotron, disaggregated P/D Pareto-dominates co-located serving across concurrency levels. (from abstract only)
- How: A flexible transfer interface generalizing across diverse cache layouts/sharding semantics and heterogeneous hardware (NVIDIA GPUs, AWS Trainium), composing with prefix caching and CPU offloading.
- Where applicable: vLLM-specific implementation but explicitly hardware-agnostic; the cross-vendor maintainer group suggests this becomes the reference interface. The hard constraint is cache-layout diversity across hybrid model architectures.