EP-Overlap: Hiding Expert-Parallel Communication in MoE Training#
PyTorch Conference 2026 digest Program Adjacent pick 20–21 October 2026
Sanket Jayant Purandare (PyTorch Distributed and Compilers), Syed Ahmed (NVIDIA, PyTorch Core)
- Why: MoE expert-parallel all-to-all is the dominant new traffic pattern on AI fabrics; knowing how much of it can be hidden by framework-level scheduling tells you how much fabric headroom a MoE fleet actually needs.
- What: Graph passes that hide EP communication by splitting each MoE layer into independent slices and overlapping token exchange with any available compute (experts, recompute, weight-gradient math), forward-with-forward and backward-with-backward. The same machinery also hides "page stashing," compacting worst-case activation memory from O(worst_case × L) to O(worst_case + actual). On DeepSeek-V3: 76–86% of EP comm hidden, ~1.3x throughput, +4% absolute MFU — and the authors call that a lower bound. (from abstract only)
- How: Implemented as graph passes in TorchTitan's GraphTrainer, working under FSDP and PP with any schedule (not just DualPipe), no model changes, bitwise-matching eager numerics — overlap rather than wire-byte reduction.
- Where applicable: DeepSeek-V3-scale MoE training; a framework-level (TorchTitan) technique that generalizes across schedules and parallelism strategies.