KV Cache-Aware Routing and P/D Disaggregation on Kubernetes: The Parts Public Benchmarks Don't Show#

Speakers: Yuchen Fama, Ashish Kamra

• Why: Agentic inference breaks the core assumption of LLM serving — the server no longer controls its cache lifecycle; the client does, through prompt construction and growing multi-turn context. That pushes prefill strategy, eviction, and routing up to the scheduler layer, which is exactly the K8s-native serving-stack problem (vLLM, llm-d).
• What: A walkthrough of the open serving stack for agentic workloads: why KV-aware routing through a fleet-wide scheduler is the first optimization (ahead of adding capacity), when prefill/decode disaggregation actually pays off versus the workload shapes that justify its complexity, plus tiered KV offload and wide expert parallelism — demonstrated on GLM-5.2 with vLLM and llm-d. Attendees get a tuning decision framework: which lever to pull first and where more GPUs don't help (from abstract only).
• How: Cache-aware routing (schedule requests to replicas by prefix reuse), separation of compute-bound prefill from memory-bound decode, tiered KV cache offload, and workload-signal-driven decisions; economics framed as prefill dominating cost with cache reuse as the primary lever.
• Where applicable: Open-source stack on Kubernetes — broadly applicable to anyone operating inference fleets, especially prefill-heavy agentic traffic. Grounded in trace replays per the abstract, but no numbers are given, so the framework's thresholds will need the talk itself.