Seeing the Plumbing: Profiling vLLM Speculative Decoding on NVIDIA Blackwell#
AI Engineer World's Fair 2026 digest Program Core pick 29 June – 2 July 2026
Speakers: Sheilah Kirui (Akamai, per linked example repo)
- Why: Speculative decoding is usually discussed in averages; this talk shows how to actually measure its engineering trade-off — guess right and you bypass the memory-bandwidth bottleneck, guess wrong and you waste compute — with live instrumentation rather than benchmark folklore.
- What: A live demo routing identical workloads through baseline and speculative vLLM configurations on a single NVIDIA RTX 6000 Blackwell GPU, profiling three vectors: time-per-output-token latency delta, the exact KV-cache/VRAM tax of tracking parallel token states within a 96GB budget, and the draft-acceptance tipping point where speculation falls below baseline efficiency. Open-source example repo (in active development) (from abstract only).
- How: Side-by-side StreamStream — a Streamlit app and a live Grafana dashboard — instrumenting the three vectors so the memory dynamics and acceptance-rate decay are visible in real time.
- Where applicable: The profiling methodology (isolating TPOT, memory footprint, and acceptance rate; forcing memory pressure) transfers to any engine/GPU pair; absolute numbers are tied to one Blackwell GPU and model pair.