GUARD: SCALABLE STRAGGLER DETECTION AND NODE HEALTH MANAGEMENT FOR LARGE-SCALE TRAINING#

guanliang liu, Abhinandan Patni, congzhu lin, et al. (affiliations not listed in the program; deployed on large-scale foundation-model pretraining)

• Why: NCCL tests and GPU burn-ins catch fail-stop, not fail-slow — and on multi-month, tens-of-thousands-of-GPU runs, silent stragglers quietly eat efficiency. This is operational health management for training fleets, and the approach transfers to any large fleet.
• What: A system combining lightweight online performance monitoring during training with an offline node-sweep mechanism that qualifies nodes before production admission. In production: up to 1.7× improvement in mean FLOPs utilization, run-to-run training-step variance reduced from 20% to 1%, higher MTTF, and reduced debugging overhead (from abstract only).
• How: Continuous workload-level performance signals (not synthetic micro-tests) detect both acute failures and long-running degradation; systematic sweeps pre-qualify nodes.
• Where applicable: The deployment is a large-scale pretraining cluster (org not fully specified in the program), so exact gains are fleet-specific. The core lesson — workload-level signal beats synthetic health checks for fail-slow detection — is broadly applicable.