Practical Network Reliability Engineering#

John Howard · Head of Network Infrastructure — Proton AG

• Why: Four years of applying NRE (SLI/SLO discipline) to a real network, with outcomes: 10x capacity since 2021, -40% per-port cost, zero extra headcount, zero out-of-hours change windows.
• What: A reframing from device health to capabilities and services: a 100G wavelength is a service; "packets DC1→DC2 within 8–10ms with N+1 redundancy" is a capability — and capabilities map cleanly to SLIs. Notably, "we missed targets early on. That was a feature not a bug — it surfaced which layer needed attention."
• How: Design contracts between layers (Edge ↔ Core ↔ Fabric ↔ OOBM) and between network and applications — e.g., apps never exceed 50% of assigned capacity (p99), the network never reduces capacity >50%, anycast instead of VRRP, default-deny VRF egress. SLIs composed in Prometheus recording rules (port state + MACsec CA + IS-IS adjacency + BGP session = DCI health); a gNMIc/NATS/Prometheus/Grafana/Alertmanager stack with Sloth for SLO burn-downs; canary rollouts and auto-isolation for residual risk.
• Where applicable: Proton-specific design (EVPN fabrics, MACsec-protected DCI wavelengths), but the capability→SLI method and the bilateral contract framing are generally applicable. Includes a hot take worth arguing with: you cannot be a network engineer and a software engineer concurrently — pick one.