Traffic at 3am: Armed with a Toothpick against AI Scrapers#

Cheryl Andrea Fernando (SURF)

• Why: AI training scrapers are now an operational cost and risk problem for anyone serving public content — and the defensive landscape is fragmenting fast. Lightning-talk depth, but a useful map.
• What: A survey of mitigation strategies: robots.txt (voluntary, widely ignored), Cloudflare's AI Labyrinth (trapping non-compliant crawlers), monetization via Cloudflare Pay-per-Crawl and Akamai's Tolbit/Skyfire, open-source countermeasures like Anubis (proof-of-work gating that makes bot traffic expensive) and tarpits (Nepenthes, Quixotic, Iocane — feeding Markov babble to poison training data, with real caveats: SEO damage and self-inflicted resource costs). The genuinely hard problem: residential proxies, which defeat datacenter-IP blocklists because the IPs are real home connections, often resold without their owners' knowledge.
• How: The promising detection thread is TLS fingerprinting (JA4+ from FoxIO, already adopted by Cloudflare and DataDome): deprecated cipher suites and odd extension combinations flag scraper clients, complementing approaches like the Amadeus research on TCP-vs-TLS round-trip-time differences for detecting residential proxy use. The recommended posture is modular — combine robots.txt, experimental setups, and fingerprinting to fit your risk tolerance.
• Where applicable: General web-serving context, NREN-specific only in the tension between openness (discoverability) and protection (research data).