Glossary · Data collection
Web crawler
A web crawler, also called a spider or bot, is a program that automatically discovers and fetches web pages — usually by following links from page to page or reading sitemaps — to build a search index, an archive or a dataset.
Reviewed 30 Oct 2026
How a crawler works
- Frontier. A queue of URLs to visit, seeded with start pages or sitemaps.
- Fetch. Each URL is checked against robots.txt and requested, with a per-host limit on concurrency.
- Parse. Links are extracted, normalised and added to the frontier; content is stored or handed to a parser.
- Deduplicate and revisit. Known URLs are skipped and changed pages are revisited on a schedule — see deduplicating listings for the content side.
Crawler vs scraper
| Crawler | Scraper | |
|---|---|---|
| Goal | Discover and fetch pages | Extract specific fields |
| Scope | Broad: whatever links lead to | Narrow: known page types |
| Output | Pages, an index, a URL graph | Structured records |
Most real systems combine both: a crawler finds product or listing pages and a web scraping step pulls the fields out.
What a polite crawler does
- Identifies itself with a descriptive User-Agent and a contact URL.
- Follows robots.txt, re-reading it at least daily.
- Limits requests per host and backs off on 429 and 5xx responses — see rate limiting.
- Caches unchanged pages and requests compressed responses.
Google's documentation draws a useful line between common crawlers, which always respect robots.txt for automatic crawls, special-case crawlers that work under an agreement with the site, and user-triggered fetchers that act for a person. AI agents that browse on a user's behalf sit closer to the last group; see AI agents.
FAQ
Questions
Something not covered? Write to contact@arcproxies.io.
Is a crawler the same as a bot?
A crawler is one kind of bot. Bot covers any automated client — monitors, chat bots, scrapers, agents — while a crawler specifically discovers and fetches pages by following links or sitemaps.
Do web crawlers need proxies?
Not to exist, but often to do their job: to see pages as they appear in another country or city, and to keep any single address within a site's per-address limits. Proxies do not entitle a crawler to more than a site will serve.
How do sites verify a crawler is who it says it is?
By checking more than the User-Agent. Google, for example, says its crawlers can be identified by the user-agent header, the source IP address and the reverse DNS hostname of that IP together.
How fast should a crawler go?
As fast as the site comfortably serves, which you learn from its responses rather than guess. Start with low per-host concurrency, honour Retry-After and back off on errors.
Sources & review
Last reviewed by the ARC Proxies team. Prices are read from the live catalog at build time.
- [1]Google for Developers: Google Crawler (User Agent) Overview
- [2]IETF: RFC 9309: Robots Exclusion Protocol
In practice