Migrating From Bespoke Hardware To An Open Load Balancing For Kubernetes#

Sergey Kolobov, Marcin Wojciechowski (Arista Networks, per slides)

• Why: Replacing a 6.4Tbps proprietary stateless L4 load balancer with MetalLB in BGP mode is exactly the BGP-native Kubernetes ingress pattern many teams are evaluating, and the talk's core contribution is the network-side fix for ECMP's nastiest gotcha: widespread connection resets when the backend set changes.
• What: A full migration story: a 4-node MetalLB cluster (dual E5-2650 v4, 20Gbps LACP per node) sustains 6–10k RPS and 30k+ established connections, and benchmarked identically to the hardware appliance (~80 vs ~79.6 req/sec, same latency, 100% 200s over 100k requests). The key technique is pre-provisioning resilient ECMP groups on the ToR (ip hardware fib ecmp resilience <IP>/32 capacity 16 redundancy 2) — fixed modulo duplication of next-hops so only flows hashing to the failed next-hop remap, instead of all flows rehashing.
• How: MetalLB/FRR mode with BFD for sub-second failover; kube-proxy conntrack headroom (32k/core → nf_conntrack_max of 1,572,864); NIC ring buffer tuning (512→8192) against microburst RX drops; ingress nodes spread evenly across racks; ArgoCD GitOps; vtysh inside the speaker pod for BGP/BFD debugging. Also candid costs: strict port exposure replaced the "widely open" old model, adding a release-review step for developers.
• Where applicable: The resilient-ECMP fix is switch-ASIC specific (Arista syntax shown; other vendors have analogues) — verify your ToR supports it before adopting. Otherwise broadly applicable to bare-metal Kubernetes with a BGP-capable fabric. The stated next steps (Cilium BGP vs MetalLB consolidation — noting Cilium lacks native BFD — eBPF CNI migration, IPv6 dual-stack) are a useful roadmap of where K8s edge networking is heading.