Network OS Upgrades at Hyperscale#

Sujitha Sekar Rajan – Meta Platforms Inc

• Why: Upgrading NOS across tens of thousands of multi-vendor devices without either security exposure or production incidents is a recurring dilemma, and this talk ships the pipeline and two rare, detailed war stories.
• What: Meta's vendor-agnostic 5-step upgrade process (vendor testing → planning → internal testing → canary → cardinality-gated rollout) with a Testbed-as-a-Service framework: tag-driven test selection on real lab hardware across all vendors and roles (50+ test cases for backbone edge alone), automated gating so no image reaches production without a full pass, progressive canary (lab metros → low-traffic → high-traffic), and cardinality rules (per-site/per-metro/global limits, freeze periods, independent role pipelines). War story 1: a vendor code refactor truncated version numbers from 64-bit to 32-bit, desyncing RIB and FIB silently under route churn (~9 days to failure at 5k updates/sec) after ~⅓ of the fleet was upgraded — only a reboot clears corrupted forwarding state, drain does not. War story 2: a re-timer DFE re-tuning bug on one vendor's odd-numbered 100GE ports produced persistent framing errors after link flaps while the interface showed up, requiring three-vendor coordination to isolate.
• How: Layered defensive pipeline; automated drain→reboot→undrain remediation; permanent forwarding-discard detectors promoted from incident tooling; shift to maintenance releases; sustained route-churn/FIB-consistency checks added to qualification after story 1.
• Where applicable: Explicitly Meta-scale (tens of thousands of devices, 3 hardware vendors, 5+ roles, up to 1.5M-prefix tables), but the pipeline design and lessons — real hardware not emulation, vendor labs can't reproduce production-scale churn, keep the detector you built during the incident — transfer to any multi-vendor operator.