COMET: Fine-grained Computation-communication Overlapping for Mixture-of-Experts#

Shulai Zhang, Ningxin Zheng, Haibin Lin, Ziheng Jiang, Wenlei Bao, Chengquan Jiang, Qi Hou, Weihao Cui, Size Zheng, Li-Wen Chang, Quan Chen, Xin Liu (ByteDance-style production-infra author roster)

• Why: MoE routing is the dominant source of bursty, hard-to-balance east-west traffic in modern AI clusters, and this paper quantifies it brutally — inter-device communication of a MoE layer can occupy 47% of total model execution time. This is the traffic pattern your fabric has to survive.
• What: An optimized MoE runtime with fine-grained computation-communication overlap: single MoE layer execution 1.96× faster, 1.71× average end-to-end speedup. Deployed in production on ten-thousand-GPU-scale clusters with claimed savings of millions of GPU hours — a real deployment claim, not just a testbed result. (from abstract only)
• How: Data-dependency analysis and task rescheduling to overlap communication with computation at fine granularity, plus adaptive workload assignment to eliminate per-rank communication bottlenecks. The communication volume itself isn't reduced — it's hidden behind compute, which changes the latency-sensitivity profile the fabric sees rather than the load.
• Where applicable: The 47% communication figure and speedups come from "popular models and frameworks" on large GPU clusters — the technique is general for distributed MoE, but the absolute gains depend on the compute-to-communication ratio of your specific model and interconnect (faster links shrink the hideable bubble). Production validation at ten-thousand-GPU scale is a strong transferability signal, but it is a single operator's stack.