Skip to content

Scaling Foundation Models: From Broken to Near-Linear Multi-Node Training#

PyTorch Conference 2026 digest Program Core pick 20–21 October 2026

Abstract

Sheng Huang — Pinterest

  • Why: A measured production case of "the network is the bottleneck" — GPUs busy waiting on cross-node traffic — with a diagnosis-first playbook that any operator of multi-node training can copy. The numbers are unusually honest (they show the broken starting point).
  • What: Pinterest's TorchRec foundation-model training went from 5x slower at 2 nodes and 1.21x scaling at 4 nodes to 3.9x at 4 nodes (97.5% of ideal) and 7.5x at 8 nodes (93.75%) — by attacking cross-node All-to-All from distributed embedding lookups, identified via PyTorch Profiler and NCCL traces. (from abstract only)
  • How: Reduce bytes on the wire (FP8 communication with FBGEMM), balanced TorchRec embedding sharding and reshaping, Distributed Checkpoint for load-time resharding, and a 2D parallel topology that keeps expensive All-to-All local and syncs replicas with AllReduce — i.e., topology designed around the slowest link.
  • Where applicable: Pinterest production on AWS (EFA networking) — org- and cloud-specific in its specifics, but the playbook (measure, cut wire bytes, checkpoint portably, design around the slowest link) is broadly transferable.