Running a 20T-Token Data Pipeline: Infrastructure Lessons from Production#

Speakers: Bogdan Gaza (Arcee, per abstract)

• Why: The systems engineering that keeps curation pipelines running on petabytes and thousands of GPUs almost never gets written down; this is a rare end-to-end production account, including the operational details (metadata stalls, multi-week fault tolerance, CPU/GPU pool fragmentation) that break naive designs.
• What: Lessons from two production pipelines — Arcee Trinity-Large's corpus scaling 8T → 10T → 20T tokens (8T+ synthetic tokens on clusters peaking at 2,048 H100s) and a 100B-token Thomson Reuters legal mid-training corpus. Concrete findings: fetching metadata from object storage across millions of files became the dominant idle-time source (offloaded to Spark plus a lightweight file-level distribution scheme, driving idle time to near zero); one-to-one Spark↔Ray partition mapping yields idempotent, resumable execution so a node failure doesn't mean reprocessing; an in-house scheduler routes CPU-heavy Spark and GPU-heavy Ray+vLLM workloads to isolated node pools; per-model vLLM tuning (batch size, speculative decoding, n-gram sampling) yielded up to 40% throughput; and the curated corpus is treated as a versioned deployment artifact (from abstract only).
• How: Hybrid Spark + Ray + vLLM architecture, idempotent partition mapping for fault tolerance, isolated node pools per workload class, and per-model inference tuning rather than stack defaults.
• Where applicable: Trillion-token-scale synthetic data generation and mid-training pipelines; org-specific in scale, but the failure modes (metadata bottleneck, long-job resumability, resource fragmentation between CPU and GPU pools) are general.