Evaluating Opsmate: Grading AI Incident Investigations at Scale#

Chinmay Gandhi, Narayanan Sankaran, Khushbu Thakur, Ankit Agarwal, Vaidyanathan PK, KC Balsu, Akash Jothi (Meta)

• Why: Everyone is deploying incident-investigation agents; almost nobody can answer "did it do a good job?" This tackles the evaluation gap for an agent handling ~1.5K incidents/day — the hard prerequisite for trusting autonomous ops.
• What: Opsmate Evals: an orchestrator- and model-agnostic framework for scoring autonomous incident investigations, combining automated ground-truth synthesis from post-incident artifacts with LLM-as-judge graders scoring 0–100 on root-cause accuracy, investigation path logic, and mitigation actionability (from abstract only).
• How: Ground truth is synthesized from chat transcripts, fix diffs, and retro docs rather than hand-curated, sidestepping the expensive-labeling problem; calibrated judge graders cover the multi-dimensional, subjective notion of "correct."
• Where applicable: General pattern for any org running incident agents — the ground-truth-synthesis idea works wherever post-incident artifacts exist. Calibration and judge reliability are the unvalidated parts; treat scores as a starting point, not ground truth.