Radius: Range-based Gradient Sparsity for Large Foundation Model Pre-training#

Mingkai Zheng, Zhao Zhang

• Why: A communication-volume play: if you can cut the bits moved per training step, you relax pressure on the interconnect that carries data-parallel gradients. Useful input for anyone doing capacity math on training-fabric uplinks.
• What: A gradient-sparsity algorithm that accelerates large-FM pre-training while preserving downstream task quality — 40% sparsity on GPT-2.0B across 64 A100s cuts per-step training time 21% and total pre-training time 19% with no downstream-task degradation. Notably shows that the obvious prior approach (top-k sparsity with AdamW) fails to converge, so the win is not free. (from abstract only)
• How: Two insights: only a small fraction of gradients contribute per iteration, and the spatial distribution of large-magnitude gradients is stable over time — so sparsity structure is maintained, avoiding the dense-communication blowup that top-k methods hit in later training phases.
• Where applicable: Data-parallel pre-training of GPT-family models; evaluated at 355M–2.0B scale on A100s. The time savings come from reduced gradient communication, so gains scale with interconnect contention. Convergence claims are pre-training-specific; not yet shown for RLHF or other finetuning regimes.