Building Cloud Infrastructure for AI#

Dave Hughes, Lukas Stockner (Stelia)

• Why: Most "GPU clouds" are really HPC clusters with a portal. This is a full-stack recipe for an actual elastic multi-tenant GPU cloud from open-source parts — including the parts vendors never discuss (PCIe P2P through the IOMMU, RDMA partitioning, firmware).
• What: An end-to-end architecture: standard OEM 8-way GPU servers, K8s used internally only (customer workloads run in containerized VMs for real isolation), ephemeral nodes imaged via mkosi + UEFI HTTP netboot, and a performance-focused storage split (local NVMe via SPDK at ~26 GB/s / 98% of raw throughput; Ceph for durable networked storage via virtio-fs/SPDK/Rook).
• How: Networking is the highlight: SONiC underlay with routing-to-host, VXLAN/EVPN overlays isolated in their own namespaces (cluster / public / per-tenant), and a custom SDN where endpoints announce prefixes via K8s CRDs that a daemon translates to BGP for hardware edge routers — IPv6-first public overlay, with per-instance port slices of a shared IPv4 address. RDMA: one NIC per GPU (3.6 Tbit/s, no sharing), InfiniBand partitioned via PKeys + GUID/VF assignment, or RoCE isolated at the switches. Virtualization: plain QEMU wrapper (KubeVirt rejected as integration-first rather than isolation-first), NUMA-local GPU/CPU selection, and PCIe topology modeled through QEMU — with the pitfall that IOMMU forces GPU-NIC P2P through the CPU root complex unless you use ATS on the NIC and ACS Direct Translation on the PLX.
• Where applicable: Anyone building or evaluating GPU cloud infrastructure. Explicitly not hyperscaler-specific — the entire point is reproducibility with open source — but it does assume you control the physical fleet and fabric.