Getting 1+ Tbps from an x86 server#

Branimir Rajtar, 5x9 Networks

• Why: A concrete, quantified datapath-tuning story: a virtualized BNG went from 160 Gbps to 1.2 Tbps on a single commodity x86 server with no specialized hardware. The optimization methodology (profiling-driven, cache-focused) transfers to any high-packet-rate software datapath — exactly the kind of datapath-internals work that is core here.
• What: A CUPS (control/user-plane separation) vBNG — dashboard, controller, and DPDK-based forwarder — was pushed through successive optimization waves: ~30% from newer hardware plus application tuning, ~30% from deep DPDK/CPU/PCIe/BIOS/NIC configuration, and ~20% from Intel vTune-guided code rewrites. The final revelation: the dominant residual cost was CPU cache misses — data structures not fitting 64-byte cache lines, plus 4K memory pages, causing constant main-memory round trips. A major cache-alignment rewrite and a shift from many small 4-vCPU VMs to one large instance closed the gap to 1.2 Tbps.
• How: SR-IOV and Intel DPDK bypass the kernel entirely (with the known tradeoff of CPU running at 100% in poll mode); vTune identifies hot spots; code is restructured to separate read/write data sections, reorder instructions for multiple execution units, use full cache lines, and force compiler optimizations. Each change was performance-tested in isolation.
• Where applicable: Results are specific to their PPPoE/IPoE BNG workload on Intel Xeon Gold with DPDK-supported NICs — treat the numbers as one datapoint, not a guarantee. The method (profile, fix cache behavior, consolidate instances) is generally applicable to anyone writing or buying software datapaths.