Running Open vSwitch on a large CPU System#

Speakers: Eelco Chaudron, Red Hat, Inc.

Why: Modern DC hosts are 128–256 CPUs, and this is a rare measured study of what actually breaks in ovs-vswitchd and the kernel datapath at that scale — including an OOM-killer incident reproduced during testing. Directly relevant to anyone running OVS on current-generation EPYC-class compute.

What: On a 128-CPU EPYC 7702P (OVS 3.3, RHEL 9.5), the revalidator process stops scaling past a handful of threads: with 200k datapath flows, mean flow-dump duration goes 169.7 ms → 180.7 ms → 183.4 ms for 4/32/128 revalidator threads, due to lock contention in nl_dump_next() (the mutex serializes netlink dump recvs to avoid an in-kernel EINVAL race on pooled sockets). With traffic landing on many CPUs, per-CPU sw_flow_stat structures (32 bytes each, allocated on demand) consume 782 MB for 200k flows at 128 CPUs and 1.5 GB at 256 CPUs — the real problem is not the memory but the latency of accumulating non-linearly-stored stats, which grows dump duration until a ~1.3 s threshold forces the flow limit down. Interface-statistics sweeps (~every 5 s) take 63 ms at 64 CPUs vs 104 ms at 128, with kernel exception counters adding ~451 ns per interface at 128 CPUs.

How: USDT probes and bpftrace-based scripts (reval_monitor.py, kernel_delay.py), veth/netns test topologies, Scapy traffic pinned with taskset, and 1024-veth-pair configurations to isolate the statistics path. A simulated pod-churn test (128 OVN ports, continuous add/delete) shows ~50% more recvmsg() time at 128 cores, traced toward netdev_get_addrs()/getifaddrs() but left for further investigation.

Where applicable: Kernel datapath (the sw_flow_stat issues are OVS kmod-specific) on large-CPU hosts — K8s nodes, NFV compute. Follow-ups listed: remove the nl_dump_next() mutex, optimize kmod stats gathering, possibly patch glibc's getifaddrs() (which wastes ~6% gathering unneeded per-CPU IPv6 stats).