Fully Adaptive Routing Ethernet in Scale-Up Networks#

Yongtao Yang (Centec), Yinben Xia, Weifeng Zhang (Tencent), Peilong Wang (Baidu), Yan Zhuang (Huawei), Xiaohu Xu (China Mobile), Zongying He (Broadcom), Nan Wang (Intel, Hygon), Jian Guo (Biren), Xiang Li (Enflame), et al.

• Why: The same WECMP idea moved into the scale-up (GPU back-end) network, where MoE expert/tensor parallel traffic is low-entropy, bursty, and collision-prone across multi-plane topologies — and the GPU itself becomes a routing participant. This is where physical fabric design and GPU interconnect are converging (OCP ESUN, AMD UALoE, Broadcom SUE-T).
• What: Extends FARE from switches to GPUs: each GPU has multiple 800G ports, at least one sub-port per scale-up plane, and performs per-flow or per-packet weighted load-balancing across planes using path-bandwidth feedback. New in -02: considerations for memory-semantic operations — weak ordering (full packet spray + GPU reorder buffers), strong ordering (in-order per QP), and partial ordering, selectable via the per-flow vs per-packet mode.
• How: Per-flow mode needs ≥1 RDMA queue pair per sub-port per GPU pair and per-flow switch balancing (ordered delivery); per-packet mode needs a single QP per GPU pair and spraying across planes (disordered delivery). Switches do matching weighted balancing.
• Where applicable: Ethernet-based scale-up networks specifically. A 64-GPU disaggregated SuperPod PoC over single-stage Clos is near completion; a 1024-GPU opto-electronic hybrid cascaded pod is under development. Co-authorship from GPU vendors (Intel, Hygon, Biren, Enflame) signals real hardware engagement, but this is early-stage (individual draft, -02).