AI Inference Without Boundaries: Dynamic Routing With Multi-Cluster Inference Gateway#

Rob Scott, Google & Daneyon Hansen, Solo.io — Adjacent

• Why: Extends Gateway API into multi-cluster GPU scheduling — a networking-control-plane answer to GPU scarcity, from the maintainers of Gateway API and the Inference Extension.
• What: Multi-Cluster Inference Gateway, a new part of the Inference Gateway project that routes inference traffic to wherever GPUs are available across clusters. (from abstract only)
• How: Builds on existing Gateway API and multi-cluster patterns to dynamically shift traffic toward clusters with free GPU capacity, for cost optimization and availability. (from abstract only)
• Where applicable: Organizations with GPU capacity spread across multiple clusters; general Gateway API-based approach.