Lightning Talk: Seeing Through the RDMA Fog: Monitoring RoCEv2 with sFlow#

Peter Phaal — InMon Corp.

• Why: AI/RDMA clusters create a genuine blind spot: kernel bypass means tcpdump, wireshark, and even eBPF-based host agents see nothing. For a PM who tracks both lossless fabrics and host-based observability (eBPF), this is the counterpoint — some visibility has to come from the switch ASIC.
• What: A demonstration, on SDSC's Expanse production cluster (93k cores, 208 GPUs, 100G hybrid fat-tree), of sFlow-based RoCEv2 monitoring: RDMA I/O operations decoded from sampled packet headers (opcode, queue pair, PSN, DMA length, virtual address), ECN/CNP congestion-message rates, RoCE ack-credit tracking, per-link queue depth and transit delay, and — critically — immediate notification of every dropped packet with where/why/who.
• How: Standard sFlow agents in leaf/spine switches stream sampled headers and counters over the out-of-band management network (no interference with RoCE); three lines of SONiC config enable it (sflow collector, sflow drop-monitor-limit 50, sflow enable), similar on Arista/Cisco/Dell/NVIDIA. sflowtool + tshark decode the InfiniBand transport headers from samples; new extensions add per-sample transit delay/queue depth and per-discard reports (reason-coded, e.g. dst_net_unknown). Ultra Ethernet is just another header format for the pipeline — no agent change needed.
• Where applicable: Broadly applicable — this is a standard switch feature, not new silicon, and the technique covers exactly the visibility gap that host tooling cannot. Drop-notification support requires current releases (202511 SONiC for drop-monitor). The one honest caveat: sampling-based, so it complements rather than replaces per-flow precision.