Can LLMs write fast multi-GPU kernels? We built a benchmark to find out.#

Speakers: Simran Arora

• Why: Production inference is bottlenecked by communication — all-reduce alone is claimed at >20% of latency on Llama-3.3-70B — yet every existing LLM-kernel-coding benchmark is single-GPU. This is the first systematic evaluation of multi-GPU kernel generation, right at the intersection of AI codegen and GPU interconnects.
• What: ParallelKernelBench (PKB): 87 problems from real codebases where the task is replacing PyTorch + NCCL with a CUDA kernel that moves data directly over NVLink. Frontier models (GPT-5.5, Gemini 3 Pro, Opus 4.7) solved under a third of problems, and fewer than a quarter of those beat the naive baseline — though a few cases produced kernels faster than anything publicly available, including one for NeMo-RL's GRPO training loop with no prior optimized public reference. Open source (from abstract only).
• How: Benchmark plus evaluation framework measuring correctness and speedup against a naive multi-GPU baseline, with data movement over NVLink directly rather than via NCCL collectives.
• Where applicable: General and open source; results are a snapshot of frontier coding models at the time of the study. Most useful to teams exploring LLM-assisted performance engineering on multi-GPU nodes.