"What can possibly go wrong?" Stories of peculiarities of modern server hardware#

Vladimir Smirnov (NVIDIA, personal hobby project)

• Why: A technical deep-dive into modern x86 server platform topology (NUMA, SNC, DDIO, PCIe lane mapping) and how to avoid catastrophic performance losses when building high-throughput network systems. Relevant to the PM's interest in high-performance networking hardware.
• What: A follow-up to the speaker's 1 Tbps/1 Gpps load generator talk, covering how PCIe slot placement, Intel Sub-NUMA Clustering, DDIO, and CPU core pinning can make a ~150 mpps difference in VPP performance. Includes a detailed analysis of AMD Epyc Zen5c CCD/IO-die topology and its fabric bandwidth implications.
• How: Breakage scenarios: VPP limited to 64 threads (mainline VPP_MAX_WORKERS); worst-case PCIe cross-connection (NICs in slots on opposite CPU dies) causing ~150 mpps loss due to DDIO/SDCI behavior across NUMA domains; re-pinning VPP workers to the correct socket recovers performance. On AMD: for pure traffic generation, CCD topology doesn't matter (fabric bandwidth 64GB/s read per CCD is sufficient) but the total machine capacity is ~1.4 Tbps.
• Where applicable: Anyone building high-performance Linux-based networking on modern Intel (Sapphire/Emerald Rapids) or AMD (Zen5) servers. The specific numbers are for VPP with ConnectX-7/BlueField-3 NICs, but the platform topology awareness is universally critical.