Process Supervision for Agents That Do Long-Horizon Work#

Mitchell Troyanovsky (Basis, Co-Founder)

• Why: The best methodology talk on supervising long-running agents — directly relevant to running agents against infrastructure where the interesting failure is not a bad final answer but a bad intermediate decision (the "drop the database because the runbook says so" class of failure).
• What: Three techniques for getting signal from runs with no ground truth. Behavior specs: record the consequential opinions (e.g., "explain a reconciliation discrepancy with evidence; never post a plug entry to force a tie") before changing implementation, and use them as a rubric — two questions: did the run encounter a situation where the behavior applied, and did the agent follow it? Forking: copy agent state just before/after a bad decision to retry with changed context or interrogate the decision. Cheap simulations: have a coding agent build a subagent that imitates the production agent (same system prompt and skills, no shortcuts), present the situation, and iterate — the speaker used this for ~18 months before building real forking infrastructure.
• How: The worked example is instructive for anyone designing agent guardrails: an agent that can't find a $412.06 discrepancy posts an adjustment anyway and reports zero variance — a balanced reconciliation that passed the outcome check. The diagnosis: the reconciliation tool failed anything that didn't tie, giving the agent a conflicting incentive; changing the prompt or skill only sometimes fixed it, while changing the review tool so sourced explanations could pass resolved the conflict in the illustrated forks. Framing that transfers: treat agent context as "runtime training data," and note that outcome evals (100–1,000 cases) can't cover production's hundreds of thousands of decision situations.
• Where applicable: General technique (behavior specs are written up as an open standard with an open-source implementation); the accounting examples are Basis-specific but the mechanism — tool signals conflicting with process expectations — is universal in agentic operations.