Glossary · Data collection
Web scraping
Web scraping is the automated extraction of specific data — prices, listings, reviews, text — from web pages into a structured form such as rows or JSON, by fetching pages with an HTTP client or a browser and parsing the content.
Reviewed 30 Oct 2026
How a scraping pipeline is built
- Fetch. An HTTP client for HTML and JSON endpoints; a headless browser only when content appears after scripts run.
- Parse. Selectors or a schema pull fields from the page; structured data embedded in the page is often the most stable source.
- Validate. Check types, ranges and empty results, so a layout change fails loudly rather than writing nonsense.
- Store. Keep every record with its source URL, location and timestamp. The web scraping with Python guide builds one end to end.
Scraping, crawling and APIs
| Approach | Best when | Watch out for |
|---|---|---|
| Official API | The provider offers the data you need | Quotas, coverage, cost |
| Scraping | The data is on public pages and no suitable API exists | Layout changes, rate limits, the site's terms |
| Crawling | You first need to find the pages | Scope creep, duplicate URLs |
Where proxies come in
Two jobs: seeing a page as it appears in a given market, through country or city targeting, and spreading requests so no single address carries a whole collection run. Neither changes what a site is willing to serve. Bandwidth adds up quickly once browsers are involved — estimate it with how much bandwidth scraping uses.
Ground rules
- Collect public, logged-out pages only; nothing behind a login, paywall or challenge.
- Follow robots.txt and the site's terms, and keep request rates modest.
- In the EU, Article 4 of Directive (EU) 2019/790 allows text and data mining of lawfully accessible works unless the rightholder has reserved it, for online content by machine-readable means. How that applies to a project is a question for counsel.
- The full rules for ARC traffic are on the acceptable use page.
FAQ
Questions
Something not covered? Write to contact@arcproxies.io.
Is web scraping legal?
It depends on the jurisdiction, the data, how it is accessed and what the site's terms say. Collecting public, logged-out pages at a modest rate is common practice; personal data and copyrighted works bring specific rules. Take legal advice for your case.
What is the difference between web scraping and web crawling?
Crawling discovers and fetches pages by following links; scraping extracts specific fields from pages you already know. Many projects crawl to find pages and then scrape them.
Do I need a headless browser to scrape?
Only when the data appears after JavaScript runs. Many sites embed the same data in the HTML or load it from a JSON endpoint, which is faster and uses far less bandwidth to fetch directly.
What is web scraping used for?
Price and assortment monitoring, travel fare aggregation, search-result tracking, review monitoring, research datasets and AI training corpora, among others — wherever the data needed is published on public pages.
Sources & review
Last reviewed by the ARC Proxies team. Prices are read from the live catalog at build time.
In practice