Skip to content

Sorry we messed up#

DENOG 17 (Essen) digest Program Core pick Essen · 9–11 November 2025

Stefan Funke

Play the recording: Sorry we messed up

  • Why: A concrete postmortem of a global route leak caused by an Arista RCF (Routing Control Functions) bug — exactly the kind of hard-won failure story that transfers across organizations. The PM's BGP and network operations interests make this directly relevant.
  • What: The story of how renaming Arista RCF functions (to encode IP version, peer IP, and session type in function names) triggered a global route leak via Arista BUG 1060542. During RCF recompilation, BGP peers shared an "undefined" RCF function state; update-group merging caused two low-scale peers to reconcile with a high-scale peer's advertised state before the withdraw-all-routes state was processed — leaking all prefixes to those peers and causing session flaps. Fixed in EOS 4.31+ maintenance releases.
  • How: The root cause is a race condition between the RCF agent compiling new function names and the BGP agent using those names for filtering. Workaround: duplicate the RCF function or map the new function to the old one, commit, wait for compilation, then update BGP config — or shut peers before renaming associated RCF functions. The talk covers the full deployment pipeline (Ansible-generated config, session-based commits, checkpoint with commit timer).
  • Where applicable: Arista EOS operators using RCF for BGP route filtering. The broader lesson — automated config changes to filtering logic can have vendor-bug landmines — applies to any operator using programmable route-policy frameworks on any vendor. Org-specific to the speaker's network (34 PoPs, ~70 devices) but the failure mode is general.