Battle of the Bots: Which AI Models Actually Work for Network Engineering Tasks#

Eric Chou — Network to Code

• Why: A genuinely reusable decision framework rather than another AI demo — and the methodology (expected-value analysis plus task-specific evaluation) transfers to any AI adoption decision, including agent/identity tooling.
• What: The premise: "which model is best?" is the wrong question; the right one is "what is the expected value of using AI for this task with this model?" A lightning talk with a decision-tree method: map decision points, calculate probabilities of success, and derive expected value per option — illustrated with a worked RCA example (AI root-cause platform at 75% success rate, 30-minute resolution vs 4-hour manual, $4k/hr downtime cost, $10k SLA penalty — EV of AI-powered RCA $11,000 vs -$26,000 for manual).
• How: Three practical evaluation methods to establish the probability inputs: expert review (human in the loop), automated code evaluation (digital twin for pass/fail), and AI-as-judge. The takeaway: build your own evaluation system and iterate — with a sensible default (Claude) as the starting point, not the conclusion.
• Where applicable: Task- and org-specific by construction — the whole point is that you run it on your tasks with your cost/downtime numbers. Broadly applicable wherever AI tooling is being considered for repeatable technical work; the evaluation-system pattern (expert + code + AI judge) is the durable part.