Going beyond show commands and screen scraping with BMP#

Bart Dorlandt (freelance network automation engineer, Netherlands)

• Why: Verifying that BGP announcements you configured actually propagated and passed import policy is a classic gap; most shops do it with screen scraping. BMP (BGP Monitoring Protocol) is the right tool and this gives the architecture and the scaling numbers.
• What: A production pipeline for pre- and post-policy BGP data: router → BMP collector → Kafka → consumer → database → API, replacing screen scraping of show commands for customer-facing verification.
• How: The initial consumer managed 30 msg/sec — nowhere near enough for millions of RIB entries — and was rewritten with a pool of async Kafka producers, reaching 45,000 msg/sec. Hard lessons: every stage (collector, consumer, DB) can be crippled by too much data, so filter early, deduplicate, and decide what you actually need; peer-down notifications are the critical code path; the BMP collector implementation used was incomplete and needed patches (Kafka TLS, Docker).
• Where applicable: Broadly applicable wherever BGP policy correctness needs verification (peering, DC fabrics, WAN). The pipeline is buildable from open components but the scaling numbers are one implementation's — treat 30→45k msg/sec as directional, not a benchmark.