Skip to content

[Almost] Fail & Tell: Stop The World#

@Scale: Systems & Reliability 2026 digest Program Adjacent pick 25 June 2026

Session recording Materials

Phil Lopreiato (Meta), Rahul Iyengar (Meta)

  • Why: A near-miss postmortem with a genuinely unusual mechanism: re-bootstrapping the infrastructure control plane of an entire region on the fly. Control-plane recovery and DR-preparedness are underrepresented topics, and this is a real production story, told by Meta's sitewide reliability people.
  • What: The March 2025 incident in which Meta avoided a prolonged sitewide outage, containing the issue in minutes by rebuilding the control plane for a whole region live using emergency tooling — presented as a case study in DR preparedness and calm incident command. (from abstract only)
  • How: Not detailed in the abstract; the substance will be in what the emergency tooling does, how regional control-plane state is reconstructed, and how the response was organized.
  • Where applicable: Meta-specific event, but the lessons — rehearsed emergency tooling, control-plane as a recoverable artifact, decisive regional action over slow diagnosis — transfer to any operator with regionalized infrastructure.