How to respect robots.txt in your crawler

How RFC 9309 matching works

A robots.txt file is a list of groups. Each group starts with one or more User-agent lines and contains Allow and Disallow rules. A crawler finds the group whose user-agent value matches its product token, case-insensitively; if none does, it obeys the * group. Within the group, every rule whose path matches the URL's path — including the query string — is a candidate, and the one with the most characters wins.

Take this file: User-agent: *, Disallow: /search, Allow: /search/about, Disallow: /*.pdf$.

The order of lines in the file does not matter for the result — which is exactly where older parsers go wrong. For the term itself, see robots.txt.

Fetching, caching and error codes

• One file per origin. The file lives at /robots.txt on each scheme and authority, so shop.example.com and example.com — or HTTP and HTTPS — each have their own.
• Redirects. RFC 9309 says crawlers should follow at least five consecutive redirects, even to another host.
• Status codes. A 4xx such as 404 means the crawler may access any resource; a 5xx, or no answer at all, means it must assume complete disallow.
• Caching. Do not use a cached copy for more than 24 hours unless the file is unreachable. Google says it generally caches for up to 24 hours as well.
• Size. The parsing limit must be at least 500 KiB; Google ignores content beyond 500 KiB.

The broader context — rate limits, terms of service and personal data — is covered in collecting travel data responsibly.

Crawl-delay and other non-standard lines

RFC 9309 lets crawlers interpret records outside the protocol, naming Sitemap as an example, but it does not define Crawl-delay. Support is therefore per crawler: Google's documentation says crawl-delay is not supported, while Anthropic's documentation for its crawlers says it supports the non-standard Crawl-delay extension. Python's parser exposes crawl_delay() and request_rate() for the lines it finds.

A respectful crawler treats a stated delay as a minimum number of seconds between requests to that host, combines it with its own per-host limit and still backs off when the server answers 429 or 503 — see handling 429 rate limits.

Parsing robots.txt in Python

urllib.robotparser.RobotFileParser offers set_url(), read(), parse(), can_fetch(), mtime(), crawl_delay(), request_rate() and site_maps(). Its documentation does not state which precedence rule it applies, and we found that it changed between releases. Running the example file above:

• Python 3.12.6, 3.13.5, 3.13.12 and 3.14.0 returned disallowed for /search/about, because the Disallow line comes first, and allowed for /docs/fares.pdf, because * and $ were not understood.
• Python 3.14.6 returned the RFC 9309 answers for every row of the table.
• Product token, not header. can_fetch() compares only the part of the string before the first slash. Passing Mozilla/5.0 (compatible; GPTBot/1.1; …) matched the * group instead of a GPTBot group in our test; passing GPTBot matched correctly.
• The read() method has its own policy. It calls urlopen without a timeout, treats 401 and 403 as disallow-all and other 4xx as allow-all, and sends urllib's default User-Agent. Fetching the file yourself, as in the code below, gives you a timeout, your own identity and the RFC's error handling.

The second code sample is a one-line known-answer test. Run it in your deployment image; if it prints False, use a parser that implements RFC 9309 rather than relying on rule order.

AI crawler user agents and opt-outs

AI companies now run several crawlers each, so site owners can allow one purpose and refuse another. The tokens below are the ones their operators document.

If you run your own crawler, give it a unique product token and a page that explains what it does, so site owners can write a group for it. If you build AI agents that browse for a user, or collect AI training data, decide explicitly which token your traffic answers to and document it. See also web crawler and user agent.