No Packet Left Behind: AWS's approach to building and operating reliable networks#

Lincoln Dale — Amazon

• Why: The reference talk on hyperscale DC fabric design and closed-loop network operations — architecture, auto-remediation, and a vendor-agnostic discard-metric standardization effort the PM should track.
• What: AWS's design tenets and numbers: cellular datacenter architecture (access cells, intra-/inter-AZ core cells) for blast-radius isolation; single-chip 1RU 12.8T fixed-port switches (32×400G) as the capacity building block — 100Tbps per rack, 3,200Tbps per 32 racks — explicitly choosing redundancy between boxes (3-tier Clos) over large-chassis dual-monolith. Operations side: 5-stage auto-remediation, >1.8Bn metrics/min, >96% of events auto-remediated; deployments always automated with intent validation (Batfish for routing-policy/ACL errors, plus an internal VM/containerlab-style system for scale validation) and rollback on any out-of-expectation metric.
• How: Key insight that auto-remediation needs (a) architecture that supports it, (b) an accurate signal of impact that indicates cause, © a deliberately small action set (isolate/restore device or link, roll back, move traffic, escalate) — because the wrong action is worse than none (draining a congested device worsens congestion). Notably, vendor discard counters were too inconsistent (grey failures, double counting), so AWS defined its own classification scheme, published as an IETF individual draft (draft-evans-discardclass), and mapped 64–256 hardware drop counters per platform onto it.
• Where applicable: AWS-specific production scale and tooling, but the tenets, the rollback discipline, and especially the discard-classification work transfer to any large fabric operator. The discard taxonomy draft is relevant to anyone doing grey-failure detection in DC fabrics.