Optimizing Deployment Configurations for LLM Inference#

Sungmin Cho, Jaewon Lee, Chunqiang Tang, et al. (Meta, serving the Llama family to ~1B monthly active users, per the paper)

• Why: Production experience from one of the largest inference fleets in existence, systematically mined for configuration lessons — the operational counterpart to NEST-style placement search, from the team living with the results.
• What: A systematic approach analyzing millions of deployment configurations across hardware (H100, H200, MI300X), parallelism strategies (tensor, pipeline, expert, context, data), and runtime choices (continuous batching vs prefill–decode disaggregation) under SLO constraints. Shares lessons on runtime trade-offs, the phase-specific nature of parallelism strategies, hardware heterogeneity, scaling behaviors, and MoE system implications (from abstract only).
• How: Large-scale design-space search with workload-specific SLO constraints on production traffic.
• Where applicable: Meta-specific fleet, workloads, and SLOs — the specific optima do not transfer. The structural lessons (phase-specific parallelism, heterogeneous hardware opportunity) do.