Large clusters for small models#

Speakers: Daniel Svonava

• Why: A wide catalog of small task-specific models is cheaper and faster than the frontier, but serving it is a different infrastructure problem: dedicated worker pools idle, request routers choke on the sheer volume of small requests, and users bring hundreds of LoRAs.
• What: How the speaker's team serves 1M tokens per second with small models: cluster architecture for maximum throughput and minimum latency simultaneously, plus "autoresearch" applied to their own inference code to support 10+ new models per week (from abstract only).
• How: The abstract is thin on mechanism — the substantive claims are the 1M tok/s figure, the joint throughput/latency cluster design, and using automated research to rewrite inference code for model-onboarding velocity. Treat the autoresearch angle as a lead to investigate, not a transferable recipe.
• Where applicable: Many-small-model / many-LoRA serving environments rather than frontier-model fleets; org-specific until the mechanisms are explained.