Skip to content

XStream: Meta's unified stream processing service#

@Scale: AI & Data 2026 digest Program Adjacent pick 17 June 2026

Abstract

Stella Kaval, Zheng Zhang, Shuo Xu, Jun Fan (Meta)

  • Why: A six-year production consolidation story at O(10) TB/s across 70,000+ pipelines — rare, honest material on unifying fragmented internal platforms and the operational lessons of keeping a managed stream service reliable at that scale.
  • What: XStream consolidates all of Meta's in-house stream processing onto one fully managed service supporting SQL, Python, and C++ authoring, with exactly-once processing, Scribe-based shuffle, ZippyDB and RocksDB+Manifold state stores, and Kappa architecture for ML feature freshness (from abstract only).
  • How: A vectorized C++ execution engine built on Velox under a multi-language authoring layer, a service management platform (Turbine), a Unified Trigger Framework for stateful processing, and CodeX decoupling user logic from engine execution.
  • Where applicable: Meta-specific stack decisions throughout; the value is the consolidation playbook and production experience (scaling, availability, efficiency lessons), which transfer to any operator running multiple parallel stream-processing systems. Velox's appearance here and in the agent-memory poster is a notable shared-substrate trend.