Multiple NICs as Encapsulation Endpoints for Large Scale GPU Cloud#

Girish Moodalbail (NVIDIA), Han Zhou (NVIDIA)

• Why: In GPU servers each GPU typically has a dedicated NIC (4 GPUs = 4 NICs), so inter-node AI training traffic is multi-homed and each NIC's IP becomes an encapsulation endpoint — while vanilla OVN assumes one encap IP per chassis. This is the scaling problem AI clouds are hitting right now, and it lands squarely in physical DC / scale territory.
• What: Multi-NIC encap endpoint support in OVN/OVN-Kubernetes: per-VIF selection of source/destination encapsulation IPs, full-mesh tunnels (all local × all remote encap IPs), new flow programming using an Encap-ID in REG13[16..31], and host-side routing fixes for both the kernel and OVS-DPDK paths. Special cases are handled: distributed gateway ports (skip pinning or fall back to default encap) and ovn-ic transit-switch ports (proposal: multiple TS-LR ports plus /32 routes and a new request-encap-ip).
• How: ovn-encap-ip carries a comma-separated IP list; per-interface encap-ip external_ids drive Port_Binding:Encap in the SB DB, populated by ovn-controller. The kernel path needs source-based policy routing plus rp_filter/arp_ignore/arp_announce sysctls; the DPDK path required an upstreamed OVS multi-table routing fix (ovs-router). Egress flows scale O(E×P) — local encap IPs × remote port bindings.
• Where applicable: Demonstrated on NVIDIA GPU cloud / cloud gaming topologies, but the mechanism is generic OVN. Scale is the open issue: 8 NICs × 1000 chassis ≈ 64k tunnel ports per node, which stresses ovs-vswitchd. Mitigations proposed but not yet landed: flow-based tunnels (loses IPSec/BFD), a "Transport Plane" grouping that cuts 64→8 tunnels per node (87.5% fewer), and intra-node tunneling to preserve HW offload across NICs.