SparkSentry: Autonomous Error Detection, Triage, and Repair for Spark at Scale#

Varun Srinivas (Meta)

• Why: This is the cleanest production example on the program of the full agentic-ops loop — detect → classify → triage → fix → review — running continuously on real fleet failures with concrete throughput and noise-reduction numbers. Anyone thinking about AIOps→autonomous-ops will want this pattern.
• What: An autonomous pipeline over Meta's millions of daily Spark jobs: twice-daily runs ingest ~10K errors, produce ~20–50 root-cause clusters, file Phabricator tasks with full investigation context, and delegate fixes to AI agents that locate the bug, write a fix with tests, and submit a diff for human review — eliminating the 7–10 duplicate tasks per error class that manual monitoring produced (from abstract only).
• How: A three-stage clustering stack — deterministic regex (36+ patterns), TF-IDF + DBSCAN for unknown errors, then LLM semantic merging (Llama 3.1-70B) — with rate-based severity detection scaled to fleet size, and a multi-layer dedup system (exact cluster ID, flexible title, normalized title with diff migration) to prevent duplicate tasks across runs.
• Where applicable: The pipeline is Meta-internal (Scuba, Phabricator), but every stage maps to standard equivalents elsewhere; the layered "deterministic first, ML for the residue, LLM for the remainder" classification and the human-review gate are directly transferable design decisions. Error-fix autonomy is bounded to the Spark codebase context.