SHIP: SRAM-Based Huge Inference Pipelines for Fast LLM Serving#

Andrew Bitar, Aravind Vayalapra, Baorui Zhou, et al. (Groq, per the paper — the deployment studied is Groq's public cloud)

• Why: A production counterpoint to HBM-based GPU serving: SRAM-resident weights eliminate the decode memory-bandwidth bottleneck, and the system runs at hundreds of billions of tokens daily. The interconnect story — a synchronous, low-diameter network scaling to thousands of chips — is a live example of topology choices driving serving latency.
• What: A review of Groq's first-generation SRAM-based inference pipelines: (1) a synchronous low-diameter interconnect enabling low-latency scaling across thousands of chips, (2) serving optimizations under tight memory capacity, and (3) large-pipeline designs that sustain efficiency and latency across varying prefill-to-decode ratios and context lengths, delivering state-of-the-art latency (from abstract only).
• How: Model weights and KV state live in SRAM spread across a chip fabric; deterministic, synchronous pipelining keeps communication off the critical path.
• Where applicable: Groq-specific silicon and interconnect — this is one vendor's architecture, not a technique you can adopt. But as a data point on how scale-up fabric topology and memory placement shape the throughput–latency frontier, it is directly relevant to anyone comparing interconnect strategies for inference.