Gray failures in AI Fabric#

Navtej Singh – Clockwork

• Why: AI training turned the DC fabric from boring plumbing into the throughput ceiling, and the failures operators now burn hours on are invisible to standard tooling — the motivating example is a GPU cloud that spent 19 hours hunting a slow training job caused by a hot transceiver whose FEC corrections had been climbing for days while the link stayed up.
• What: A working taxonomy of gray failures in AI fabrics — optic degradation via rising FEC corrections, intermittent CRC below thresholds, asymmetric paths, ECMP hash polarization landing collectives on hot links, PFC headroom exhaustion, sub-poll-interval microbursts — plus three fully worked scenarios (ECMP polarization, optic degradation, PFC cascade) showing what each telemetry layer sees and what it misses.
• How: Three time-aligned layers: host (NCCL kernel timing, RDMA QP stats, per-rank step time), edge (PFC pause counters, queue depth, per-packet HW timestamps), fabric (gNMI/INT streaming, buffer occupancy, FEC). The proposed gap-closers are sub-microsecond PTP clock sync (software-only, no special silicon) plus an out-of-band probe mesh (<1% of link capacity, separate QoS class) for one-way delay and per-path attribution; histograms over gauges; alert on p99.9 percentile drift; the SLO becomes p99.9 collective step time.
• Where applicable: Aimed at RoCEv2/lossless GPU-training fabrics and deliberately vendor-agnostic (IEEE 1588, OpenConfig/gNMI, INT/P4, NCCL profiler, DCGM, Linux RDMA counters). The speaker works for a vendor (Clockwork), but the scenario walkthroughs and tool references are open and portable. Cardinality discipline is flagged as the main scaling constraint.