Native Tiered KV Cache Offloading in vLLM: From Storage Offloading to Disaggregated Serving#

Or Ozeri — IBM

• Why: Long-lived agentic sessions turn KV-cache preservation into a storage/I-O tiering problem with idle sessions holding state for hours or days — the memory-side complement to the KV-transfer talk, and a new consumer of fabric bandwidth.
• What: vLLM's native tiered KV-cache offloading framework, newly integrated upstream with no external dependencies. The key design choice: all transfers route through CPU memory as a universal transport hub. (from abstract only)
• How: Routing through host memory yields minimal GPU overhead (a simple copy), consolidated I/O from a single CPU buffer, hardware-agnostic operation with no specialized transfer APIs, and independence from KV cache memory layout (which varies across hardware, attention backends, parallelism schemes, and model architectures).
• Where applicable: vLLM deployments generally — the design is deliberately layout- and hardware-agnostic; deployment and extension guidance promised.