KV Cache in the Agentic Era: The Five Forces that Decide Where the Cache Lives, and What a DPU Can Do About It - Jigar Kuverji Savla, Tensordyne#

Speakers: Jigar Kuverji Savla, ASIC Architect & Product Manager, Tensordyne

• Why: Agentic AI workloads re-read entire conversation histories each turn, which turns KV-cache placement into a systems problem spanning chip memory, system RAM, local drives, network storage, and — the speaker argues — the DPU as the data mover. This is a concrete case of AI workload pressure reshaping data-path design, with numbers attached.
• What: A framework of "five forces" that determine where KV-cache memory should live: (1) agent turns — move memory blocks only at turn boundaries; (2) execution stages — keep the main processor on control while the DPU moves whole blocks; (3) model sizes — size storage to actual context needs and stop moving data at the crossover point; (4) orchestrators — store shared text once, route tasks to the holder, track hit rates by tier; (5) hardware paths — the physical route determines real speed. Claims measured results: a well-chosen path yields a 3.4x speedup while a poor one costs a 3.84x slowdown, raising the question of whether freed chip memory should buy cheaper servers or more useful capacity. The talk offers a blueprint for measuring the whole data path. (from abstract only)
• How: Tiered memory placement with turn-boundary migration, DPU-driven bulk data movement, orchestrator-level deduplication and locality-aware routing, and an end-to-end path-measurement methodology.
• Where applicable: Specific to agentic inference serving (context-heavy, read-mostly, turn-structured workloads) rather than training or general serving. The 3.4x/3.84x figures come from the speaker's own testing at Tensordyne; methodology details are promised in the talk but not in the abstract, so treat the exact numbers as single-vendor until seen.