Resilient AI Supercomputer Networking using MRC and SRv6#

Christoph Paasch (OpenAI) and others

Why: This is the single most load-bearing talk of the program: production experience of running RDMA over a 75K-GPU fabric at OpenAI and Microsoft, and the clearest public statement yet of how frontier training clusters actually do path selection and failure recovery.

What: Three coupled pieces — MRC, an RDMA transport that sprays a single QP's packets across all fabric paths and actively load-balances; two-tier multi-plane Clos topologies (4 x 200G or 8 x 100G planes) that let >100K-GPU clusters stay two-tier; and static SRv6 source routing so the endpoint, not the fabric, decides the path.

How: Regular RoCE pins a QP to one hash-selected path, so flow collisions leave links idle while collided flows drive tail latency — and tail latency directly sets step time in synchronous training. MRC keeps one logical QP for the application but sprays across path selectors, each PS resolving to a complete SRv6 destination address that names plane/NIC port and T0 uplink. Loss is treated as path feedback: bad PSs are swapped from a backup set, converging to <1 loss/s/NIC within minutes with no pre-populated denylist. Switches stay dumb — static config, no resilient ECMP hashing, no BGP convergence — and a clustermapper loopback SRv6 probe feeds a denylist without needing a peer. Production incidents shown: a T1 that stopped forwarding hit ~25% of QPs and ~580K dropped packets before MRC routed around it; four flapping 200G links cost ~25% throughput for about a minute, with QPs staying alive.

How much is deployed: MRC is implemented by three NIC vendors — NVIDIA (ConnectX-8/9), AMD (Pollara, Vulcano), Broadcom (Thor Ultra) — on 400/800G RDMA NICs.

Where applicable: Explicitly a large-operator design: it assumes a purpose-built multi-plane Clos, source-routing-capable NICs, and MRC silicon. The transferable ideas — endpoint-driven path selection, treating loss as path health, keeping switches static — apply well below that scale. Packet trimming and ECN load-balancing were mentioned but not covered.