Routing LLM Inference in Production: From Engine Signals to Policy#

Speakers: Qianru Lao, Lu Zhang

• Why: Request-level routing across engines, clusters, and geographies is emerging as its own infrastructure layer — an inference load balancer — which is the GPU-era analog of load-balancing architecture the PM already knows well.
• What: A generalized Inference Load Balancer (ILB) proxy/controller architecture: a low-latency proxy applies routing weights and request-path signals, while a controller computes source-cluster-to-engine weights from demand, capacity/performance profiles, replica state, and geography cost. Includes the operational debugging patterns AI engineers actually need: reading engine signals, explaining why a request landed on a given backend, handling retries and load shedding, and keeping routing observable (from abstract only).
• How: Two-tier split — data-plane proxy for request-path decisions, control-plane controller for weight computation — with engine-signal feedback loops. The abstract notes the content is framed generically "without exposing OpenAI-specific internals," which strongly suggests a large-lab production origin.
• Where applicable: Multi-cluster, multi-engine inference fleets; the architecture pattern is general, but expect transferable patterns rather than a deep operator postmortem, since the speakers deliberately abstract away their production specifics.