You Want to Upgrade WHAT? A Field Guide to Risky Network Changes#

Nadia Pinaeva (NVIDIA), Dumitru Ceara (Red Hat), Patryk Diak (Red Hat)

• Why: A rare, detailed production migration narrative: OVN-Kubernetes' legacy Layer2 UDN topology was redesigned around the OVN Transit Router while customers ran on it, with real failure modes and fixes documented — exactly the incident-detail-rich content worth a PM's time.
• What: The legacy L2 topology (node-specific gateway routers) broke VM live migration (stale ARP for the migrated gateway MAC; IPv6 RAs from multiple node gateways creating broken multipath defaults, worked around by iptables-blocking external RAs in every L2 pod), doubled maintenance for each new feature, and forced EgressIP hacks. The new topology makes a transit router the network's default gateway with the same IP and MAC on every node. The bulk of the talk walks the rolling upgrade through a half-old/half-new cluster, including hand-crafted routes and dummy IPs to bridge old and new gateway routers during the mixed state, and hand-crafted route advertisements to flush stale routes.
• How: Rolling per-node upgrade rewires topology while workloads are drained and rebooted; NAT moves through the transit router; explicit static routes (e.g., 100.65.0.2 via 100.88.0.5) stitch the two topologies together.
• Where applicable: OVN-Kubernetes Layer2 UDNs in production; the authors note it is based on actual joint debugging sessions across the OVN and OVN-Kubernetes communities. The migration playbook generalizes to any non-disruptive OVN topology change.