From Lab to Production: The Hard Parts of Agentic NetOps#

Tobias Bühler - NetFabric

• Why: The follow-up to SwiNOG #41's agentic-AI-in-network-ops talk, now with real production deployment scars. For a PM thinking about agentic identity and AI safely operating production networks, this is the most concrete failure-mode catalog available — all drawn from Swiss ISP/MSP deployments on 1000+ devices.
• What: The three hard problems of production agentic NetOps — correctness, security, and model/paradigm churn — with real failure cases and code-level fixes. The core thesis: rely on code wherever it matters, and run continuous evaluation, because prompt and skill changes alone do not reliably change agent behavior.
• How: Correctness: agents analyzing firewall policies checked only the PREROUTING chain or the first 50 of 950 rules, ignored default-drop policies, and mis-evaluated subnet membership — fixed with a dedicated, multi-vendor firewall logic module in code (given rules and a flow, does it pass: yes/no). Security: agent-written SQL against a multi-tenant metrics DB is wrapped in guardrails that rewrite queries at the AST level to enforce tenant isolation, cap analyzed rows, block modifications, and reject oversized results before they blow the context window; CLI access is mediated by a grammar of available commands scraped from lab devices, categorized with operator approval, so production lookups only pass approved commands (a DMVPN troubleshooting example drops from ~25 blind failed commands to one-shot success). Evaluation: pitfalls include a TODO-list prompt change that tripled run time (3→10 min), synthetic test traces without production noise, and unit-style tests that don't predict long agentic investigations. Reported results: ticket root cause and proposed fix in 2–3 minutes in multiple deployments, early hardware-degradation detection, config-vs-source-of-truth drift detection.
• Where applicable: Any operator considering AI agents on production networks. The patterns (guardrails in code, approved command grammars, eval harnesses fed by production incidents) are general; the specifics reflect NetFabric's own deployments and products.