MCCL: Fault-Tolerant Collective Communication for Large-Scale GPU Training#

Ben Carver — Meta

• Why: Dynamic communicator membership — shrinking and growing as ranks fail or recover — is the collective layer learning what routing protocols learned decades ago about membership churn. Pairs directly with torch.distributed.reconfigure() and the TorchPass work.
• What: MCCL, Meta's open-source fault-tolerant C++ collective library: hardware failure treated as a first-class design principle, with dynamic communicator reconfiguration that doesn't interrupt the training job. Production-grade AllReduce and Send/Recv via TorchComms integration, with claimed performance parity with NCCL. (from abstract only)
• How: Redefines the communication lifecycle around failure and recovery instead of static membership and full-job restarts.
• Where applicable: Aimed at tens-of-thousands-of-GPU fleets where failure is routine; open source, so broadly inspectable — but the parity claim is from the abstract, with numbers presumably in the talk.