Skip to content

Five Bottlenecks, Five Fixes: Systematic Performance Diagnosis for PyTorch Distributed Training#

PyTorch Conference 2026 digest Program Adjacent pick 20–21 October 2026

Abstract

Paulo Aragao, Aravind Neelakantan — AWS

  • Why: A repeatable diagnostic methodology for the question you get paged about — "is it the network?" Case 3 in particular (0.4% network utilization with MoE expert parallelism turning out to be a pipeline bubble, not a network problem) is exactly the misattribution pattern that wastes fabric-engineering time.
  • What: Five real case studies mapped onto a decision framework (compute-, memory-, communication-, or I/O-bound), each with specific torch.profiler signatures, NCCL debug patterns, and targeted fixes. Headline results: 83x data-loading gain (1.2% → usable GPU utilization), 28% throughput recovered from pipeline stages, and 6.8x end-to-end improvement on a production autonomous-driving model. (from abstract only)
  • How: Structured profiling → classify the bottleneck → apply the matched fix; delivered as a repeatable checklist for any distributed training workload.
  • Where applicable: Workload-agnostic methodology from AWS architects; the case studies are customer-specific but the framework and profiler signatures transfer cleanly.