Network State Checking: A Case Study in Automation#

Craig Henderson — REANNZ

• Why: A model case study in migration-verification automation with an unusually honest evaluation — including the incidents the tooling failed to catch and its structural limitations.
• What: During a national hardware refresh (~97% generated configs, 8–12 member handovers per 3-hour nightly window), REANNZ built tooling to snapshot all per-interface state pre/post-migration — link/MTU/optical RX, VLAN push/pop, EVPN multi-homing forwarding/blocking status, BFD session state, BGP neighbors with routes received/accepted/advertised, OSPF adjacencies — and diff it so the engineer's attention goes to the most-likely-broken services. The "trial" section is the gem: three real incidents (omitted BGP domain attribute causing loop detection to suppress active routes; omitted MTU isolating a remote region; omitted downstream BGP config), each with a verdict — tooling innocent, tooling available-but-unused, or no tooling ("terrible engineer"). Stated limitations: the tool only knows what it knows (unanticipated config flies under the radar), network-as-source-of-truth doesn't scale for service metadata, and the approach is network-centric, not service-centric.
• How: The "Bender" automation monorepo (unified config parsing since 2019, generator library for standard configs) feeding pre/post state capture and diffing; next step is an intent database plus an orchestration layer owning service lifecycle, with custom config managed inside the orchestrator for visibility.
• Where applicable: REANNZ NREN context (member handovers, perfSONAR) is org-specific, but the pattern — automated state-diff verification plus honest post-hoc attribution — transfers to any refresh/migration project, DC or otherwise.