Context Parallelism for Scalable Million-Token Inference#

Amy Yang, Jingyi Yang, Aya Ibrahim, Xinfeng Xie, Bangsheng Tang, Grigory Sizov, Jongsoo Park, Jianyu Huang (Meta / NVIDIA collaboration per author roster)

• Why: This is the paper at the intersection of long-context inference and datacenter interconnect reality. It directly answers a fabric-planning question: how much inter-host bandwidth does million-token prefill actually need, and does it need RDMA or does TCP suffice?
• What: A context-parallelism scheme for long-context LLM inference that achieves near-linear scaling of prefill latency up to 128 H100 GPUs across 16 nodes — 1M-token prefill with Llama3 405B in 77 seconds at 93% parallelization efficiency and 63% FLOPS utilization; 128K context in 3.8s. (from abstract only)
• How: Two lossless exact ring-attention variants — pass-KV and pass-Q — covering full prefill, persistent-KV prefill, and decode. Ring attention turns the inter-node fabric into a circulating KV/Q pipeline, so the bandwidth demand is per-link and steady rather than bursty all-to-all.
• Where applicable: The headline empirical result for a fabric PM: benchmarks on H100 hosts interconnected with both RDMA and TCP show similar scalability, i.e. the method works over medium-to-low inter-host bandwidth common in commercial datacenters. Ring attention does impose a fixed, latency-sensitive communication pattern across nodes, so node placement and hop count matter. Evaluated on H100 clusters; the RDMA-vs-TCP equivalence is the transferable finding, not the absolute latencies.