Two Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story#

Speakers: Asaf Gardin, Yuval Belfer (AI21, per abstract — Jamba models)

• Why: Distributed inference systems fail silently — wrong answers with perfect confidence, no stack trace, no sanitizer warning — and this is a rare debugging postmortem that teaches the instrumentation methodology, not just the war story. Promoted from adjacent: silent-failure debugging in distributed inference is core ops content for anyone running vLLM.
• What: Two production bugs from building Jamba models: (1) a request misclassified during scheduling loads stale state from a previous prompt-cache slot and confidently generates gibberish once every ~1,000 prompts; (2) logprob spikes during RL training that looked like training instability until they were noticed tracking rollout count, then cache size (from abstract only).
• How: Instrumenting vLLM to thread request IDs through the forward pass; building comparison scripts that expose logprob divergence; forcing memory pressure to surface rare bugs; shrinking a distributed RL training mystery into a reproducible single-script failure; and the generalizable lesson — search for variables that change failure structure rather than magnitude.
• Where applicable: vLLM-based serving and RL training loops specifically, but the instrumentation techniques and correlation-driven debugging method transfer to any cache-based serving system.