Routing Stateful AI Workloads in Kubernetes#

Maroon Ayoub, IBM & Michey Mehta, Red Hat — Adjacent

• Why: A layered, concrete treatment of why stateless K8s routing fails for LLM inference and how KV-cache-aware load balancing changes the picture — grounded in the multi-vendor llm-d effort.
• What: Routing strategies for stateful LLM workloads, from round-robin up to full KV-cache-aware load balancing, with guidance on when each applies. (from abstract only)
• How: Based on llm-d (Google/IBM Research/Red Hat) using the Gateway API Inference Extension; covers session-aware routing for long-context traffic and global vs local cache indices. (from abstract only)
• Where applicable: LLM inference platforms on Kubernetes; general technique tied to the Inference Extension.