How Meta Scaled AI Training Storage via Data Normalization#

Sarang Masti, Weiran Liu (Meta)

• Why: Training-data storage is becoming a first-order cost driver as sequence-based and multimodal models inflate feature sizes 10–100x. For anyone planning storage and data-path capacity for training clusters, this is a concrete, quantified answer to the duplication problem.
• What: A co-designed data and infrastructure change that cut storage cost 10x for Meta's largest recommendation training feature (long user sequences) by eliminating feature duplication in training samples (from abstract only).
• How: Features are moved out of the per-request training rows (where a popular content feature could be duplicated a million times) into a high-performance indexing store, with model access-pattern-aware pushdown so the trainer fetches features at read time instead of materializing them per sample.
• Where applicable: Meta-specific production setup today, but the normalization-vs-duplication tradeoff and access-pattern pushdown idea transfer to any large-scale ML training data pipeline. The gains depend on read patterns being predictable enough to push down; highly random access would erode the win.