Building AI Clusters: What Network Engineers need to know#

Brian Walsh — Nokia

• Why: AI training clusters are now the dominant forcing function in DC fabric design — rail-optimized topology, RoCE/lossless-adjacent behavior, and elephant-flow traffic patterns. squarely in the PM's RDMA/lossless-fabrics and topology interests.
• What: A structured primer mapping the AI stack to network engineering decisions: programming frameworks (PyTorch DDP, NCCL/RCCL collectives) → parallelism strategies (data/pipeline/tensor) → collective operations (all-reduce, all-gather, send/recv) → the resulting NIC traffic pattern: periodic synchronized bursts, a few elephant flows, and job completion time bound by the worst-case tail latency across the cluster — fundamentally different from steady cloud traffic. In-node is NVLink/NVSwitch (or Infinity Fabric); across nodes it's InfiniBand, RoCEv2, and the emerging UEC. Cluster build-out is modular "Scalable Units": e.g., 256 air-cooled B200 GPUs in 9 racks, or a liquid-cooled GB200/NVL72 SU of 576 GPUs across 10 racks with 4 Ethernet GPU rails, NVLink switch trays, and CDUs — pre-validated at line rate, with fixed cabling/routing/software per SU and scale-out via spine/core.
• How: Walks the full stack (CUDA kernels → collectives → topology) and shows concrete rack-elevation layouts per SU variant, with SU node count tied directly to GPU-server NIC count and leaf radix. Rail-optimized design: each NIC homed to its own rail-leaf.
• Where applicable: Vendor-educational (Nokia) but not a product pitch — the GPU/collective/topology content is vendor-neutral. Directly applicable to anyone designing or evaluating AI back-end fabrics; assumes Ethernet/RoCEv2 as the baseline with IB/UEC as context.