DreamDDP: Accelerating Low-Bandwidth Geo-Distributed LLM Training with Layer-wise Partial Synchronization#

Zhenheng Tang, Zichen TANG, Junlin Huang, Xinglin Pan, Rudan Yan, Yuxin Wang, Amelie Zhou, Shaohuai Shi, Xiaowen Chu, Bo Li (affiliations not listed in the program)

• Why: Geo-distributed training is the workload that will stress (and justify) DC-to-DC WAN interconnects. This paper quantifies how much of that WAN bottleneck can be clawed back on the ML side — directly relevant to capacity planning for AI-over-WAN before you buy bandwidth.
• What: A training framework that decouples model synchronization layer-wise: instead of synchronizing the whole model every N iterations (Local SGD) or every iteration (S-SGD), only a subset of layers is synchronized per iteration. Delivers 1.49×–3.91× speedups over leading baselines on 32 GPUs across ResNet-18/50, GPT-2, and Llama-2, with convergence rates theoretically comparable to S-SGD (from abstract only).
• How: Layer-wise partial synchronization enables overlapping parameter sync with computation without extra GPU memory; fine-grained layer-wise profiling drives the communication/computation schedule.
• Where applicable: Low-bandwidth geo-distributed data-parallel training. Evaluated at modest scale (32 GPUs) — frontier-scale behavior is unproven. It is an ML-side technique that complements, not replaces, WAN capacity planning; its value to a network PM is as a signal of what traffic geo-distributed training will and won't generate.