ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments#

YOUHE JIANG, Fangcheng Fu, Xiaozhe Yao, Taiyi Wang, Bin CUI, Ana Klimovic, Eiko Yoneki

• Why: LLM serving in heterogeneous cloud environments — mixed GPU types and, critically, diverse network bandwidth conditions — is the realistic deployment case for anyone not operating a homogeneous in-house cluster. The scheduling algorithm here explicitly reasons about network bandwidth as a placement constraint.
• What: An LLM serving system for heterogeneous clouds: a scheduling algorithm that optimizes the deployment plan under heterogeneous resource and network-bandwidth conditions, plus a lightweight re-scheduling mechanism that adapts to node failures and workload shifts without restarting services. Up to 2.1× throughput (1.7× average) and up to 2.5× better latency-deadline attainment vs. state of the art at the same price budget. (from abstract only)
• How: Network- and resource-aware deployment planning, then live re-sharding/reallocation of serving capacity as conditions change — the re-scheduling-without-restart piece is the operational lesson, since costly restarts are what usually kill adaptive placement in production.
• Where applicable: Cloud deployments with mixed GPU generations and variable inter-host bandwidth; also validated in homogeneous in-house environments. The headline numbers are cloud-price-relative, so they translate as "cloud can beat in-house at equal budget" rather than absolute performance claims.