AI Training Parallelism, From a Network Engineer's Point of View#

Adel Iazzag – NVIDIA

• Why: The best "why does my AI fabric traffic look like this" explainer on the program — parallelism strategy maps directly onto flow patterns, incast, and fabric design decisions. Presented as vendor-neutral from four years of building/running AI fabrics.
• What: A tour of the four parallelism types with their wire signatures: data parallelism (few very large AllReduce gradient flows per step, sawtooth uplink utilization); pipeline parallelism (point-to-point stage handoffs, latency-sensitive, util graphs look fine while the job waits); tensor parallelism (constant small AllReduce every layer that must stay in scale-up — if you can see it on the fabric, placement is wrong); MoE (AllToAll, many flows, uneven by destination, worst-case incast). Real jobs stack all of these on the same wires.
• How: NCCL picks ring vs tree algorithms based on topology and message size (256 GPUs: 510 ring steps vs 16 tree steps); design guidance: rail-optimized (NIC k to leaf k), 1:1 subscription, lossless with congestion control tested for many-to-one, and ECMP alone is not enough — packet or flowlet spraying needed.
• Where applicable: General technique for anyone building or operating AI backend Ethernet fabrics; the "test the collective, not the parallelism" and "give the scheduler the topology" instincts transfer regardless of vendor.