Sparing Strategies to Minimize Reliability Impact On Large Training Jobs#

Kevin Quirk, Matthew Lennie, Ehsan K. Ardestani, et al. (Meta, per the paper's framing around Meta's AI clusters)

• Why: Sparing — pre-allocated spare compute that replaces failed components — directly trades capex against goodput on multi-month training runs. Closed-form sizing guidance for that tradeoff is rare and welcome.
• What: An analytical framework with closed-form expressions for choosing sparing strategy — compute block size, number of spare blocks, spare GPU trays — cross-validated by simulation, and applied in Meta's hyperscale infrastructure to inform fault-tolerance, downtime, and goodput decisions (from abstract only).
• How: Analytical modeling of failure/replacement dynamics, plus a simulation component for validation; yields first-order production recommendations.
• Where applicable: The framework is generally formulated, but the optimal sparing parameters are strongly fleet-specific (Meta's failure rates, tray topology, job sizes). Use the method and the first-order rules, not Meta's numbers.