Skip to content

Sorry we messed up#

NetUK2 (The Brewery) digest Program Core pick The Brewery · 7–8 July 2025

Stefan Funke (Inter.link)

Play the recording: Sorry we messed up

  • Why: A route leak that leaked "everything to everyone" is exactly the class of BGP incident every operator dreads, and this one was triggered by a seemingly innocuous change (renaming RCF functions) interacting with an Arista behaviour. Postmortems like this are the highest-value content at an operator event: the lesson transfers to anyone using vendor scripting/RCF for config automation.
  • What: A postmortem of a global route leak at a transit provider: the motivation for the change, how configurations are rolled out at Inter.link, why renaming Arista RCF functions produced unexpected behaviour and a global leak, and how it was detected and fixed (from abstract and program page summary; the speaker notes it is deliberately high-level, aimed at a general NOG audience).
  • How: Narrative postmortem: intended change → rollout path → surprise behaviour → blast radius → fix. The interesting mechanism is the interaction between operator-managed RCF function names and vendor handling of them.
  • Where applicable: The specific bug is Arista EOS/RCF-specific, but the transferable lessons — "cosmetic" config changes are not cosmetic, function naming indirection can silently change policy semantics, and rollout safeguards need to cover refactors — apply to any network with scripted configuration generation. Inter.link's global transit footprint made the blast radius unusual; smaller networks leak less far but the same way.