Eywa: Automating Model-Based Testing using LLMs#

Rajdeep Mondal, Rathin Singha, Todd Millstein, George Varghese (UCLA), Ryan Beckett, Siva Kesava Reddy Kakarla (Microsoft Research)

• Why: Model-based testing finds real protocol bugs but has always been gated on humans hand-writing RFC-conformant models — months of expert effort. Eywa removes that gate and immediately found 16 previously-undiscovered bugs in DNS/BGP/SMTP implementations. This is the clearest signal yet of LLMs changing core network-engineering practice (spec → model → tests), with results that survived review.
• What: A protocol-testing framework that uses LLMs to automatically build models of intended protocol behavior from RFCs, blogs, and natural-language sources; case studies across DNS, BGP, and SMTP found 33 unique bugs, 16 novel despite extensive prior manual testing.
• How: LLM-synthesized models are combined with mitigation for hallucinations and the inability to monolithically generate complex models (decomposition, validation against spec behavior), then fed into standard model-based test generation to drive high-coverage conformance testing of real implementations.
• Where applicable: Protocol conformance testing broadly — DNS, BGP, and SMTP demonstrated; the approach is protocol-agnostic in principle where written specs exist. Artifact evaluated (available/functional/reproduced). If you own protocol conformance or interop testing, this is the paper to read this cycle.