Self-Healing PyTorch Training Through Live GPU Migration#

Pavneet Ahluwalia, Gavin Cohen — Microsoft Azure (product) and Clockwork Systems

• Why: Link flaps and node failures are the physical-fabric failure modes you live with; this talk is about the job layer's counter-move — live migration instead of checkpoint-restart. The structural diagnosis (every step ends in a collective needing all ranks; a communicator can't be rebuilt around a missing peer) is the clearest statement of why training jobs are so brittle to fabric events.
• What: TorchPass, a training-failover system presented jointly by Together AI and Clockwork Systems: on link degradation it reroutes traffic; on GPU/memory/component failure it live-migrates the affected rank's device and communicator state to a spare and resumes from the same step. Their claim: a single checkpoint-restart failure burns 45–90 minutes of wall-clock across every GPU in the job. Live demo with injected link flaps and GPU failures; a decision framework for when live migration beats checkpoint-restart, plus a TCO framing for recovered goodput. (from abstract only)
• How: End-to-end system — monitoring, health checks, failover, rank-state migration — keeping a multi-node job in continuous forward progress.
• Where applicable: Vendor joint presentation with a product and TCO calculator attached; the companion demo talk reports a published TorchPass benchmark at Llama 4 Scout scale on 64 H200 GPUs, independently validated by SemiAnalysis. The mechanism assumes spare capacity and their product; the accountability/verification framing (does a failure resume from the same step, does throughput ever hit zero, can the claim be verified?) transfers to any fabric.