Skip to content

Fix First, Investigate Later: When an eBPF Rollout Brought Down Our Network#

KubeCon + CloudNativeCon North America 2025 digest Program Core pick 10–13 November 2025

Recording

Zain Malik, Exostellar & Grzegorz Głąb, Whatnot — Core

Play the recording: Fix First, Investigate Later: When an eBPF Rollout Brought Down Our Network

  • Why: A genuine incident post-mortem with hard numbers (throughput collapse from ~800 MB/s to ~250 MB/s, packet loss spikes) caused by a silently rolled-out eBPF DaemonSet — the kind of failure-mode honesty the program mostly lacks.
  • What: An outage investigation that traced mysterious packet loss to an unannounced cloud-provider eBPF monitoring agent (Retina). (from abstract only)
  • How: Incident narrative from alerts through diagnosis to discovery of the unfamiliar eBPF DaemonSet; emphasis on how to triage when the cause is infrastructure you didn't deploy. (from abstract only)
  • Where applicable: Managed Kubernetes where the provider can inject eBPF agents; general cautionary lesson on eBPF programs contending on the datapath.