Taming AI Infrastructure Failures with Agentic Debugging#

Phillip Liu (Meta)

• Why: NCCL watchdog timeouts are the canonical "everything looks fine until a collective hangs" failure of distributed PyTorch training — anyone running GPU clusters has paid this tax. A root-cause taxonomy built from a year of internal incidents and shared with the PyTorch community is immediately actionable.
• What: A categorization of the major root-cause buckets behind NCCL watchdog timeouts (investigated at Meta, in partnership with the PyTorch community), distilled into a practical decision tree and runbook to compress triage time — plus early results using agents to assist root-cause analysis. The abstract openly flags that details are still TBD. (from abstract only)
• How: Systematic bucketing of observed failure modes into a decision tree; agent-assisted RCA layered on top of the structured workflow (a companion article describes it as "a technical, repeatable workflow").
• Where applicable: Broadly applicable — this failure mode affects anyone running PyTorch distributed training, not just Meta. Since NCCL timeouts are usually a symptom (slow node, fabric congestion, misconfigured transport, process hang), expect the taxonomy to span host, NIC/transport, and network layers.