Skip to content

Scaling Batch Graph Processing to Trillions of Edges#

@Scale: AI & Data 2026 digest Program Adjacent pick 17 June 2026

Abstract

Sahil Gandhi, Arneish Prateek (Meta)

  • Why: Graph jobs at billions of nodes/trillions of edges (LLM pretraining, recommendations, dedup) stress exactly the distributed-systems pathologies — memory blow-up, skewed shards, network overhead, gang scheduling — that any large-scale batch infrastructure has to engineer around.
  • What: Meta's new batch graph processing engine for iterative graph algorithms at trillion-edge scale, designed around reducing cost across data storage, graph partitioning, network communication, execution, and distributed coordination (from abstract only).
  • How: The abstract names the constraint set and a pluggable architecture accommodating diverse distributed graph processing models; mechanism-level detail (partitioning scheme, coordination design) is not given in the abstract.
  • Where applicable: Graph-workload specific; the network-overhead and skew-handling concerns will resonate with anyone operating large sharded data systems. Thin abstract — worth checking the poster or recording for the actual partitioning and coordination design.