Skip to content

Ship Fast, Break Nothing: How AI Tests What AI Builds#

@Scale: Product 2026 digest Program Core pick 28 October 2026

Abstract

Sophie Zeng (Meta), Sreeja Thummala (Meta)

  • Why: A production look at "AI validating AI-built code" at extreme scale — LLM-driven test generation and LLM assessors running inside the quality gate across hundreds of millions of lines of code. Relevant to anyone building or evaluating agentic dev tooling.
  • What: Meta's Just-in-Time (JiT) Catching Test system: AI autonomously generates tests designed to fail, surfacing bugs before code lands, and LLM-based assessors decide which catches deserve human attention. Deployed across hundreds of millions of lines of code, claimed to proactively prevent serious production failures. (from abstract only)
  • How: Two-stage mechanism — code-change-aware generation of adversarial "catching tests" that are constructed to expose latent bugs, followed by LLM assessors acting as the human-attention filter so the volume of machine-generated catches doesn't swamp engineers.
  • Where applicable: Depends on Meta monorepo scale (hundreds of millions of LOC) and deep CI integration; the pattern of adversarial test generation plus LLM triage is general and reproducible at smaller scale, but the effectiveness numbers cited are Meta-specific.