Unleashing Scalable Context Parallelism for Foundation Models Pre-Training via FCP#

Yilong Zhao, Xiaonan Nie, Kan Zhu, et al. (affiliations not listed in the program)

• Why: Context parallelism over variable-length training data wastes fabric capacity — over-sharding short sequences, or imbalanced bin-packing of long and short ones. FCP relaxes the rigid-ring assumption, which is a scheduling/communication co-design relevant to anyone provisioning collective fabrics.
• What: A flexible context-parallelism paradigm that shards and schedules sequences at block-level granularity, achieving near-linear scalability to 256 GPUs and 1.13×–2.21× improvement in attention MFU (from abstract only).
• How: Block-level sharding enables arbitrary peer-to-peer communication topologies (not just rings) and bin-packing of blocks from short and long sequences for balanced, communication-efficient placement.
• Where applicable: Foundation-model pretraining with heterogeneous sequence lengths. Evaluated at up to 256 GPUs — mid-scale by today's standards — so frontier-scale behavior is extrapolated.