Glossary · Data collection
robots.txt
robots.txt is a plain-text file at the root of a website, /robots.txt, in which the operator tells automated crawlers which paths they may and may not fetch, grouped by crawler name; the Robots Exclusion Protocol behind it dates from 1994 and was standardised by the IETF as RFC 9309 in September 2022.
Reviewed 30 Oct 2026
user-agent: ExamplePriceBotallow: /products/disallow: /checkout/user-agent: *disallow: /searchdisallow: /account/How the rules are read
- Rules sit in groups headed by one or more
user-agentlines. A crawler uses the group whose name matches its product token, or the*group if none does. - Within a group,
allowanddisallowlines are path prefixes. The most specific match wins, meaning the rule with the most octets. - If an allow and a disallow rule are equally specific, RFC 9309 says the allow rule should be used.
- Paths not matched by any rule are allowed.
When robots.txt cannot be fetched
| Response for /robots.txt | What a crawler does under RFC 9309 |
|---|---|
| 2xx | Parse the file and follow it |
| 4xx (unavailable) | May access any resource on the site |
| 5xx or network error (unreachable) | Must assume complete disallow |
| Cached copy | Should not be used for more than 24 hours, unless the file is unreachable |
| Very large file | Parsers must handle at least 500 KiB |
What robots.txt is not
RFC 9309 states that the rules are not a form of access authorisation: they are a request, not a lock. They also do not replace a site's terms of service, which can say more, or ARC's own acceptable use rules.
In EU copyright law, Article 4(3) of Directive (EU) 2019/790 lets rightholders reserve their works from text and data mining "in an appropriate manner, such as machine-readable means" for content published online. robots.txt is one widely used machine-readable signal; whether a given file amounts to such a reservation is a legal question, not one a glossary can settle.
Following it in practice
Fetch robots.txt per host before crawling, cache it for no more than a day, and check every URL against the group for your User-Agent token. Proxies change none of this: the rules apply to your web crawler whatever address it comes from. A step-by-step version is in respecting robots.txt.
FAQ
Questions
Something not covered? Write to contact@arcproxies.io.
Is robots.txt legally binding?
RFC 9309 itself says the rules are not access authorisation. Their legal weight depends on jurisdiction and context, for example as a machine-readable reservation under EU text and data mining rules, so treat them as binding in practice and take legal advice for specific cases.
What does Disallow: / mean?
It asks the crawlers in that group not to fetch any path on the site. Under the * group it applies to every crawler without a more specific group of its own.
Does robots.txt apply if I use a proxy?
Yes. The rules address the crawler, identified by its product token, not the network address it comes from. Changing the exit IP changes nothing about what the site has asked.
How often should a crawler re-read robots.txt?
RFC 9309 says a cached copy should not be used for more than 24 hours unless the file cannot be reached. Re-fetching once a day per host is the usual pattern.
Sources & review
Last reviewed by the ARC Proxies team. Prices are read from the live catalog at build time.
In practice