Skip to content

Achieving AI Cluster Reliability at Scale: Lessons in Reducing Workload Interruptions#

OCP Global Summit 2026 digest Program Adjacent pick 12–15 October 2026

Abstract Recording

Bob Wang (Meta), Ramachandra Reddy Kallam (Meta)

Play the recording: Achieving AI Cluster Reliability at Scale: Lessons in Reducing Workload Interruptions

  • Why: Not a networking talk per se, but network events are one of the correlated signals — and the root-cause methodology (attributing every interruption across host, firmware, network, and workload telemetry) is reusable by anyone operating large fabrics.
  • What: A framework for diagnosing AI workload interruptions at 32K→100K+ GPU scale, built on root-cause classification via Perfetto time trees correlating host telemetry, firmware logs, network events, and workload signals; key contributors include NVLink failures, HBM fragmentation, thermal failures, silent data corruption (SDC), and firmware bugs (from abstract only).
  • How: Data-driven root-cause analysis feeds back into next-gen hardware requirements, including vendor decisions on error reporting, fault isolation, and resilience; the diagnostic framework is being offered to the broader OCP community.
  • Where applicable: Meta GBx00 clusters — org-specific failure data, but the methodology (correlated time-tree attribution) and the hardware-requirements feedback loop generalize well.