Lessons Learned Testing Network Automation Solutions#

Ivan Pepelnjak · ipSpace.net

• Why: The most transferable engineering talk of the conference — the testing methodology behind netlab, plus a multi-vendor NOS bug catalog that anyone testing across platforms will hit.
• What: ~100 transformation tests plus ~100 error tests comparing against YAML-stored expected results (diffs reviewed in PRs); 25+ integration suites, 250+ individual tests across ~30 device types, requiring 3,700+ lab instances and days of runtime. Hard lessons: testing distributed systems takes longer than planned and vendor implementations will surprise you.
• How: Multi-platform testing is solved by not parsing show output on ~20 platforms — instead, FRRouting probe containers attached to the device under test run JSON show commands and validate protocol behavior indirectly. Tiered CI: pre-commit (linting, some tests) → push → PR (all transformations) → dedicated server (full integration before release).
• Where applicable: Broadly applicable to anyone building or testing multi-vendor automation. The quirks list is directly useful: Junos advertising wide IS-IS metrics as 63, Aruba CX invalid VXLAN IP checksums, mostly-broken OSPF NSSA implementations, divergent EVPN/ARP defaults, unresolved VRRPv3 checksum and MPLS entropy-label disagreements.