Skip to content

Precompile for Training#

PyTorch Conference 2026 digest Program Wildcard pick 20–21 October 2026

Abstract

Bob Ren; Aaron Orenstein (Meta)

  • Why: A genuinely surprising distributed-systems failure mode: in nominally identical SPMD programs, per-rank compilers and autotuners can diverge — choosing different kernels on different ranks, causing NCCL hangs, numerical inconsistencies, and hard-to-debug job failures. Anyone who has chased a "one rank hung" incident will recognize the shape of this problem.
  • What: Precompile for Training, an abstraction that compiles a distributed SPMD program once and reuses the resulting graph artifact across all ranks — guaranteeing identical execution and kernel selection cluster-wide, moving compilation ahead of GPU allocation, and eliminating redundant compile work across thousands of GPUs. (from abstract only)
  • How: All ranks share one compiled graph artifact; compilation is decoupled from GPU provisioning so idle-GPU-during-compile waste disappears.
  • Where applicable: Large SPMD training fleets, vendor-agnostic in principle; short talk with abstract-level detail — the rank-divergence problem statement is the durable takeaway.