Photon: Federated LLM Pre-Training#

Lorenzo Sani, Alex Iacob, Zeyu Cao, Royson Lee, Bill Marino, Yan Gao, Wanru Zhao, Dongqi Cai, Zexi Li, Xinchi Qiu, Nic Lane (Oxford / Cambridge / UCL-style FL collaboration)

• Why: A counter-current to the "ever-fatter fabric" narrative: this trains 7B-parameter LLMs from scratch across weakly connected, geographically dispersed GPU silos over the internet. If viable, it changes WAN/interconnect economics for training — less dependence on a single specialized DC, compute sourcing that follows electricity prices, and far lower inter-silo bandwidth requirements.
• What: The first complete system for federated, from-scratch, end-to-end LLM pre-training. Trains models up to 7B federatively with perplexity better than centralized pre-training; communicates 64×–512× less than baselines and beats baseline distributed-training wall-time by 35%; converges twice as fast as DiLoCo and is robust to data heterogeneity across silos. (from abstract only)
• How: Cross-silo federated learning: local training with small client batch sizes at extremely high learning rates, with periodic federated averaging — the surprising claim is that FedAvg's robustness to hyperparameters is what unlocks the aggressive data efficiency. Communication happens only at synchronization rounds, which is where the 64–512× reduction comes from.
• Where applicable: Cross-silo settings with internet-class (not datacenter-class) links between participants. Data heterogeneity and non-IID silo datasets are explicitly addressed. The result is pre-training only — fine-tuning and RLHF stages aren't covered in the abstract. Most relevant to anyone thinking about geo-distributed training capacity and WAN procurement; it lowers the interconnect bar rather than raising it.