Are LLM Performance Benchmarks Reliable?#

Speakers: Ashok Chandrasekar, Jason Kramberger

• Why: Public inference benchmarks disagree wildly — vLLM vs. SGLang numbers, analyst reports — and often fail under real-world conditions; measurement hygiene matters to anyone making capacity or engine decisions from published data.
• What: Root causes of inconsistent LLM serving benchmarks: single-process load-generation tools constrained by the Python GIL, model-level settings like temperature, undisclosed load-generation parameters (QPS, concurrency), and insufficient observability into the benchmark clients themselves. Plus Inference Perf, an open-source multi-process utility for reliable stress testing of production stacks (from abstract only).
• How: Multi-process load generation with explicitly reported parameters and client-side observability, so the load generator is not the bottleneck being measured.
• Where applicable: Universal — anyone benchmarking inference stacks; the pitfalls (GIL-bound clients, unreported concurrency) apply to every serving comparison, published or internal.