"What can possibly go wrong?" Stories of peculiarities of modern server hardware#

Vladimir Smirnov (NVIDIA, personal project)

• Why: A follow-up to the speaker's 1 Tbps/1 Gpps load generator talk, covering how modern server platform topology (PCIe lane mapping, Sub-NUMA Clustering, DDIO, CCD/IO-die fabric) can silently cost ~150 mpps in VPP performance if misconfigured. Relevant to anyone building high-throughput Linux networking systems.
• What: Concrete breakage scenarios on Intel Emerald Rapids and AMD Epyc Zen5c: (1) VPP limited to 64 worker threads (mainline VPP_MAX_WORKERS, no configurable patch merged), (2) worst-case PCIe cross-connection (NICs in slots attached to opposite CPU dies) causing ~150 mpps loss due to DDIO/SDCI behavior across NUMA domains, (3) core pinning to the correct socket recovering performance. On AMD: Zen5c CCD topology doesn't matter for pure traffic generation (fabric bandwidth 64GB/s read per CCD is sufficient), but the whole machine caps at ~1.4 Tbps generating capacity.
• How: Diagnosis via lstopo topology visualization, Intel PCM memory counters, and VPP statistics. The talk shows how to verify PCIe-to-CPU-die mapping from motherboard/CPU vendor diagrams before assuming slot layout is benign.
• Where applicable: Anyone building high-performance Linux networking (VPP, DPDK, custom packet processing) on modern Intel Sapphire/Emerald Rapids or AMD Zen ⅘ servers. The VPP_MAX_WORKERS limit is a mainline-VPP-specific constraint. Home-lab hardware but the platform topology lessons are universal.