Lessons From Scaling BPF To Detect RDMA Device Drivers Bugs In Real Time#

Prankur Gupta – Meta Platforms, Inc

• Why: RDMA driver bugs are a top cause of AI training failures at Meta — 10–20% of daily AI job crashes are RDMA-associated (not network-related), wasting 32k+ GPU hours over six months — and the RDMA control path is famously opaque to existing observability. This sits squarely at the intersection of eBPF and lossless-fabric operations.
• What: RDMATracer, an eBPF-based tool that detects RDMA device driver bugs in real time by tracing kernel state mismatches across the ibverbs control path. In production it cuts diagnosis from 10+ minutes to seconds, auto-classifies issues, and was used to triage a black-box vendor driver bug (a dma_resv_wait_timeout overflow in ib_umem_dmabuf_map_pages on a newly deployed NIC driver).
• How: fentry/fexit BPF probes on kernel functions along ibverbs ioctl paths (fexit measured at ~72% of kretprobe overhead); static analysis to prune call paths (skip reads, pure-kernel internals, cleanup/refcount functions; per-vendor focus); a ~256KB ring buffer (~8k records) exports error context to userspace; per-CPU counter maps track invocation/error/errno counts. Periodically captures "good" traces and compares against failing ones to find the bifurcation point. Measured tax: p50 88.7% lower, p99 98.9% lower bpf_tax; no impact on workflow QPS.
• Where applicable: Built for Meta's Linux/RDMA GPU training fleet (vendor-specific driver paths baked into the pruning rules), but the engineering pattern — probe-point selection via static analysis, fexit over kretprobe, ring-buffer error-context export, good-trace diffing — transfers to any operator doing eBPF-based kernel-state tracing. Directly relevant to anyone running RoCE fabrics for training.