Lessons from scaling BPF to detect RDMA Device Drivers Bugs in real time#

Prankur Gupta, Maksim Samoilov (Meta)

• Why: RDMA driver bugs are a top killer of AI training jobs — 10–20% of Meta's daily AI job crashes are RDMA-related (syscall errors), wasting 32K+ GPU-hours in six months, 5K+ attributable to the RDMA subsystem. This is the eBPF observability answer, with measured production overhead.
• What: RDMATracer, an eBPF-based framework that detects RDMA driver bugs in real time by tracing the opaque control path between verbs ioctls and the NIC, then differring live traces against known-good ones. Diagnosis time dropped from 10+ minutes to seconds.
• How: Since all RDMA interactions map to one syscall, they use static analysis to prune the traced call graph (skip reads, pure-kernel internals, refcount/cleanup functions; focus per-vendor driver calls). They chose fexit over kretprobes (~72% of the overhead), shared maps with a ~256KB ring buffer for error context, per-CPU counters for syscall/error/errno tracking, and a reactive "collect on error" design. Production impact: bpf_tax p50 down 88.7%, p99 down 98.9%, no measurable QPS change.
• Where applicable: Any large RDMA fleet (Meta-scale, but the static-analysis pruning idea generalizes to other opaque syscall interfaces). Notably it triaged a bug in a black-box vendor driver that dmesg alone couldn't localize.