Skip to content

Lessons Applied Building a Next-generation AI Proxy#

KubeCon + CloudNativeCon North America 2025 digest Program Adjacent pick 10–13 November 2025

Recording

John Howard, Solo.io — Adjacent

Play the recording: Lessons Applied Building a Next-generation AI Proxy

  • Why: From an Istio TOC/Steering member, this is the clearest articulation of why AI traffic breaks traditional proxies and what an AI-optimized data plane needs — relevant to where Envoy/Gateway data planes are heading.
  • What: The design rationale behind a new CNCF AI proxy built as part of kgateway. (from abstract only)
  • How: Argues that intelligent request batching, model-aware load balancing, and token-based rate limiting are now table stakes that Envoy-era proxies weren't built for, and walks through the tradeoffs of a purpose-built AI data plane. (from abstract only)
  • Where applicable: AI/LLM inference gateways; conceptual/architectural, broadly applicable to proxy and gateway designers.