TCP and RDMA are Killing Inference Throughput; Homa can Fix It#
AI Engineer World's Fair 2026 digest Program Core pick 29 June – 2 July 2026
Speakers: John Ousterhout (keynote; affiliation not listed in program)
- Why: Inference traffic is shifting from bulk transfers to huge numbers of tiny, latency-critical messages (agentic workflows, disaggregated KV stores), and the transports underneath — TCP and RDMA — were not built for that mix. This is the one genuinely transport-layer talk in the set, sitting squarely in the RDMA/lossless-fabric and AI-fabric intersection.
- What: An argument that legacy transports are the throughput bottleneck for modern inference workloads due to poor congestion control and head-of-line blocking, plus an introduction to Homa, which cuts short-message latency by ~10x on AI-datacenter-like workloads (from abstract only).
- How: Homa uses receiver-driven flow control (grants) and, critically, capitalizes on priority queues in switches — SRPT-style scheduling of messages into switch priority levels so short requests don't queue behind large transfers.
- Where applicable: Datacenter fabrics where switches expose usable priority queues (Homa is designed around programmable queue assignment, not deep buffering). Note the framing comes from Homa's own author, so expect advocacy; the failure analysis of TCP/RDMA on mixed-size traffic stands on its own. Broadly relevant to anyone building disaggregated inference/KV fabrics.