Computer Use at the Edge of the Statistical Precipice#

Speakers: Pierluca D'Oro

• Why: A methodology bombshell: a 1MB replay script that blindly executes a recorded action sequence — never observing the screen — outperforms frontier computer-use agents on prominent static benchmarks. If the PM consumes any agent benchmarks, this changes how to read them.
• What: The replay script's expected success rate is proven exactly equal to the source agent's pass@k in deterministic environments — meaning static CUA benchmarks largely measure memorized trajectories, not interactive competence. The talk traces this to two root causes (unprincipled environment design; naive aggregation and pass@k misuse for stateful UI tasks), proposes the PRISM design principles, and instantiates them in DigiWorld: 15 realistic sandboxed mobile apps evaluating agents across 3.2 million verified unique configurations, plus an aggregation framework that correctly handles benchmarks' nested structure (from abstract only).
• How: Formal analysis of pass@k semantics in deterministic stateful environments; sandboxed, automatically verified environment construction; nested aggregation accounting.
• Where applicable: CUA evaluation specifically, but the pass@k-aggregation critique applies to any stateful, multi-step agent benchmark — broadly relevant to how the field measures agents.