Debugging Stupid Things: If it only was DNS for once...#

Tobias Fiebig, TU Wien

• Why: The program's purest "hard-won lessons" talk: weird corner-case bugs, the debugging process used to find them, and the belief systems that got in the way — exactly the kind of content that transfers across environments.
• What: A tour of production corner cases, including a PMTUD failure where downloads for users on one transit path crawled because ICMP "need to frag (mtu 1420)" messages were being lost or ignored while other paths were fine; Microsoft treating ED25519 DKIM signatures as a syntax error while RSA validated; FRR treating 'extended-nexthop' as 'unnumbered' and emitting rogue Router Advertisements onto the AMS-IX exchange LAN; and solar-panel power monitoring causing random shutdowns whose reconnect pattern triggered a null-pointer crash in OpenBGPD. No DNS packets were the root cause in any case.
• How: Reproduce, then PCAP ("PCAP or didn't happen"), then work outward from what the packets actually say — with each case paired by a lesson about questioning the assumptions ("why one has not yet understood what one told said system").
• Where applicable: Universal; the methodology and the humility travel to any network or systems team.