December 2025

robots.txt is a Suggestion, Not a Boundary

The robots.txt standard (RFC 9309) is a voluntary protocol with no enforcement mechanism. A Disallow directive is a polite request.

What We Found

Major search engines had near-perfect compliance. Zero violations from verified Googlebot in six months. AI training crawlers were different — several requested blocked paths at rates suggesting they weren't checking robots.txt at all.

The most interesting category: crawlers that fetched robots.txt, parsed it, and systematically visited every Disallowed path. The file became a treasure map.

Practical Implications

robots.txt is appropriate for managing crawl budget. It is not a security control. And the Disallow directive itself makes an excellent tripwire.