SAKURAONE: An Open Ethernet–Based AI HPC System and Its Observed Workload Dynamics in a Single-Tenant LLM Development Environment#

Fumikazu Konishi, Yuuki Tsubouchi, Hirofumi Tsuruta (SAKURA Internet Research Center)

• Why: A top-100 TOP500 system built on a fully open networking stack — 800 GbE, SONiC, RoCEv2 — is the strongest public existence proof yet that vendor-neutral Ethernet fabrics scale to frontier LLM training. For anyone evaluating SONiC/white-box options for AI clusters, this matters more than another InfiniBand case study.
• What: A 100-node managed HPC cluster (8× H100 per node, 2 PB all-flash Lustre) on a rail-optimized 800 GbE leaf–spine fabric with RoCEv2 and SONiC, ranked 49^th in ISC 2025 TOP500 (33.95 PFLOP/s HPL, 339.86 PFLOP/s FP8 HPL-MxP) — the only top-100 system using a fully open network stack. The paper also reports single-tenant workload dynamics: small jobs dominate by count, a few large jobs consume most GPU-hours, and usage shifted from large-scale training to mid-scale iterative refinement over the project's life (from abstract only).
• How: Built on the SAKURA KOKARYOKU PHY bare-metal GPU platform; rail-optimized leaf–spine topology; open control and data plane end to end.
• Where applicable: The fabric architecture and its measured scalability are broadly relevant to Ethernet-for-AI decisions. The workload observations are specific to one research project's exclusive use of the cluster — treat them as a single-tenant data point, not a general cloud workload distribution.