Rethinking Network Troubleshooting: Smarter Tools and Better Visibility at Nexthop#

David Brown — Nexthop

• Why: An honest walk through monitoring blind spots — especially optics — and a buildable open-source visibility stack. The optics lesson alone is worth the PM's time.
• What: A "typical fault process" case: intermittent loss, everything looks fine, eventually reproduced with sustained pings — root cause a marginal optic (RX -18.86dBm, a 10km optic inside a DC). Hard-won lesson stated bluntly: "optics die a slow death and TX monitoring lies all the time," and monitoring is too inward-facing. The response: ping-all-the-things mesh (8.8.8.8 the most-pinged target, 1.1.1.1 second), a flow pipeline of pmacct (sFlow+BGP) → Kafka → ClickHouse → Grafana correlated with Telegraf/SNMP, and a custom Python daemon (PingMonitorAgent) running natively on Arista EOS via the agent framework — Kafka-driven ping targets, fping processes per source interface/IP/VRF, results keyed by router_id and if_index so they join cleanly with sFlow data, with ProcMgr auto-relaunch on crash.
• How: EOS agent framework for in-switch active probing; pmacct/Kafka/ClickHouse for per-AS loss/jitter dashboards. Includes an Arista-as-Linux exploration (network namespaces, iperf on the switch CPU: 237Mbps, then 1.21Gbps after stripping iptables and control-plane protection — a lab demo, not a recommendation).
• Where applicable: Nexthop is a small Australian DC/hosting operator; the pmacct pipeline and RX-power alerting are broadly adoptable. The in-switch agent portion is Arista EOS-specific. Good source of adoptable tooling patterns rather than strategy.