How much bandwidth does web scraping use?

What a web page weighs

HTTP Archive crawls millions of sites every month and publishes the results in its annual Web Almanac. The 2024 page-weight chapter, measured in October 2024, gives these medians — bytes transferred, so already compressed where the server compresses:

The component medians do not add up to the page median — each is the median of its own distribution — but the proportions are the point: the document you want is under 1% of what a browser downloads.

HTML-only versus rendered scraping

An HTTP client such as Python requests or Scrapy downloads exactly the URL you ask for. A headless browser downloads the document and then everything it references — scripts, styles, fonts, images, trackers — and runs the scripts, which fetch more.

Use a browser when the content is built client-side and you cannot find the JSON the page loads. Otherwise fetch HTML or that JSON directly. Many pages that look dynamic load their data from a single public JSON call you can request yourself.

A worked example: 100,000 pages

Arithmetic in decimal units (1 GB = 1,000,000 KB). The blocked-images row is a rough subtraction of medians, not a measurement. Blocking JavaScript as well is usually impossible when you rendered precisely because of JavaScript.

On per-GB pricing, the method is the bill: the same 100,000 pages cost two orders of magnitude more rendered than fetched. The pricing page prints the current per-GB price per pool.

What the medians do not include

• Your pages are not the median page. The Almanac describes the web at large; a product page or a fare search result can be heavier or lighter. Measure your own targets.
• Retries and failures use bandwidth too. A 10% retry rate is 10% more traffic.
• Redirects, cookie banners and interstitials add requests you did not plan.
• Protocol overhead — request headers, TLS handshakes on new connections — is small per request but adds up across millions of small fetches.
• Compression only helps if your client asks for it with Accept-Encoding; most HTTP clients do by default, some do not.

How to measure your own number

• Fetch 50 representative URLs per target with the method you will use in production.
• For HTTP clients, sum the response body sizes plus headers; for browsers, sum transferred bytes across all responses in the page's network log.
• Take the median and the 90th percentile, multiply by planned volume and add a margin for retries.
• Repeat when a target redesigns — page weight drifts upwards over time.

To bring the number down, see how to reduce proxy bandwidth. For a sense of real workloads, the web scraping and price monitoring pages work through monthly cost examples.