Your Change Deserves a Dress Rehearsal: Digital Twins from Production State with Containerlab and OTG#
NANOG 98 digest Program Core pick 19–21 October 2026
Brad Tarno – The Walt Disney Company
- Why: The most directly adoptable tooling talk of the program: a self-service lab-as-a-service that builds disposable twins from what production actually runs, closing the "valid config is not a safe config" gap (their motivating change: one peer-reviewed line silently suppressed 36→1 routes). Every load-bearing component is open source.
- What: A platform that harvests real device configs, transforms them per-platform to boot on virtual NOS images (cEOS, XRd, vJunos, N9Kv, FRR/gobgp), deploys via containerlab, and validates with Open Traffic Generator traffic (snappi), emulated BGP peer twins replaying real routes with full attributes, and Batfish+hier_config differential analysis of the predicted post-change config. Verdicts are graded per-axis (syntax by the real NOS parser, static, control plane, traffic, propagation) with convergence-aware traffic grading (PASS / PASS* transient loss quantified / FAIL permanent loss) and explicit INCONCLUSIVE outcomes with declared fidelity limits (MACsec stripped, multicast data plane, QoS queuing not graded).
- How: 1-hop LLDP neighbor expansion keeps twins small enough to build in minutes; peer twins are gobgp nodes injecting real sessions with full attributes (a full table via RIPE RIS MRT replay costs ~1.6KB/route, ~2.1GB resident for 1.4M prefixes); real traffic matrices replayed from Kentik top-N at lab-safe rates; the platform has no write path to production and holds no device credentials.
- Where applicable: Multi-vendor production network (Disney); fully replicable with open-source parts by any operator with config collection and lab servers. Limits are honestly declared: virtual data planes can't emulate MACsec, optics, or hardware timing.
