Experience Productionizing and Operating GB200 clusters#

Ankur Srivastava (NVIDIA), Sachin Kumar Lakharia (NVIDIA), Douglas Wightman (NVIDIA)

• Why: Rack-scale NVLink systems (GB200 NVL72: 72 GPUs with an exascale-scale NVLink fabric in one rack) change how clusters must be scheduled and operated — fabric locality becomes the dominant scheduling constraint. First-party operational experience from NVIDIA on bringing up and running a large cluster is hard to get elsewhere.
• What: Production experience operating large GB200 clusters: topology-aware scheduling in Slurm to maximize NVLink fabric utilization, GPU-occupancy and scheduling recommendations validated with a Slurm simulator, and the unglamorous bring-up work — health checks, rack testing, slow-node identification, supporting large-scale training runs. Cites >2.6x MLPerf training improvement attributable to the NVL72 system. (from abstract only)
• How: Topology-aware Slurm scheduling (placing jobs to exploit intra-rack NVLink bandwidth) plus simulation-driven occupancy policy; operational practices for node/rack validation before admitting capacity to the pool.
• Where applicable: Tied to NVIDIA GB200 NVL72 silicon and Slurm specifically — the concrete recommendations don't port directly to non-NVL systems or Kubernetes schedulers. The method (topology as a first-class scheduling input, simulator-tuned occupancy, systematic slow-node quarantine) generalizes to any tightly-coupled accelerator fabric.