Robust LLM Training Infrastructure at ByteDance#

Zuquan Song (ByteDance Seed) — Talk, Wednesday, March 18

• Why: A real production number — 97% effective training time over a three-month, 9,600-GPU run — attached to a described fault-tolerance system; strong adjacent content on training-robustness engineering.
• What: ByteRobust, a GPU infrastructure management system for robust LLM training that handles NaN values, CUDA errors, and job hangs, achieving large-capacity fault tolerance, prompt fault demarcation, and data-driven localization; deployed in production with 97% effective training time ratio on a three-month training job across 9,600 GPUs. (from abstract only)
• How: Leverages LLM training's parallelism and workload characteristics for fault tolerance and fault localization via a data-driven approach.
• Where applicable: ByteDance's production platform; the metrics are real but transferability depends on how much of the demarcation/localization method is shared in the talk.