Guide · Scraping practice
How to reduce proxy bandwidth when scraping
The biggest bandwidth saving is not rendering at all: fetch the HTML or the JSON a page loads, which is usually a small fraction of what a browser downloads. Where you must render, block images, fonts and media. Then make every remaining request cheaper — ask for compression, use conditional requests so unchanged pages return an empty 304, and cache what does not change. Each step is independent, and together they routinely cut traffic by an order of magnitude or more.
Reviewed 30 Oct 2026
The short answer
- On the median desktop page, the HTML is 18 KB of 2,652 KB. Fetching the document instead of rendering the page is the largest single saving.
- In a browser, block images, fonts and media by resource type. Images alone are 1,054 KB on the median desktop page.
- Send
Accept-Encoding: gzip, br; Brotli and gzip compress text responses substantially. - Conditional requests with
If-None-MatchorIf-Modified-Sincelet an unchanged page return 304 with no body. - Do not fetch what you already have: cache static resources and deduplicate URLs before they reach the proxy.
Techniques ranked by typical impact
| Technique | What it saves | Trade-off |
|---|---|---|
| Fetch HTML or JSON instead of rendering | Everything but the document — over 99% on a median page | Needs the data to be in the HTML or a public JSON call |
| Block images, fonts and media in the browser | Images are ~40% of median desktop bytes | Rarely breaks extraction; check lazy-loaded content |
| Block third-party scripts and trackers | Part of the 613 KB of median JavaScript | Can break pages that need a script to render data |
| Ask for compression | A large share of every text response | None — just make sure the client sends the header |
| Conditional requests (ETag / Last-Modified) | The whole body when nothing changed | Only where the server supports validators |
| Local cache and URL deduplication | Repeat fetches entirely | Needs a cache-invalidation rule |
Shares are from HTTP Archive's 2024 medians; your targets will differ. See how much bandwidth scraping uses for the full breakdown.
Fetch the data, not the page
Open the target in a normal browser with developer tools and watch the network tab while the content appears. On many modern sites the listing, price or availability arrives in one JSON response from a public endpoint the page calls without login. Requesting that endpoint directly with an HTTP client such as httpx gives you structured data at a fraction of the bytes.
Stay with what a logged-out visitor's browser requests anyway. An undocumented endpoint is still the site's property and covered by its terms — check them, and keep your request rate polite.
If you must render, block what you do not need
Browser automation libraries let you intercept requests before they leave. In Playwright, page.route can abort requests by URL pattern or by resource type, so images, fonts and media never cross the proxy. Block by type rather than by extension; many images are served without one.
Test extraction after blocking. A few pages wait for an image or font event before revealing content, and some lazy-load data on scroll.
Make each request cheaper: compression and conditional requests
- Compression. HTTP clients advertise supported encodings with
Accept-Encodingand the server labels the response withContent-Encoding(RFC 9110). Brotli (RFC 7932) and gzip both compress HTML and JSON well. Most clients send this by default; check yours. - Conditional requests. Store the
ETagandLast-Modifiedof each response. On the next fetch sendIf-None-MatchandIf-Modified-Since; if nothing changed the server returns 304 Not Modified with no body (RFC 9110). For monitoring jobs that re-check the same URLs, this can remove most of the traffic. - Caching. Follow
Cache-Controlfor static resources (RFC 9111) and keep a local cache keyed by URL. A crawler that fetches the same image or script twice is paying twice.
Stop waste before it reaches the proxy
- Deduplicate URLs after normalising query parameters; tracking parameters create endless variants of one page.
- Fetch sitemaps or listing APIs to discover URLs instead of crawling navigation pages repeatedly.
- Cap retries and back off on 429 and 5xx — a retry storm is the most expensive failure mode. See handling 429 rate limits.
- Route static, cheap pages to the cheapest pool. On ISP proxies per-GB prices are lower than on residential, which matters at volume.
Code
Worked example
from playwright.sync_api import sync_playwrightBLOCK = {"image", "media", "font"}with sync_playwright() as p: browser = p.chromium.launch(proxy={ "server": "http://proxy.arcproxies.io:8000", "username": "USERNAME_pool-arcfast_country-de_session-k3f9a1x2_sesstime-10", "password": "PASSWORD", }) page = browser.new_page() page.route("**/*", lambda route: route.abort() if route.request.resource_type in BLOCK else route.continue_()) page.goto("https://example.com/products", wait_until="domcontentloaded") print(page.title()) browser.close()import json, pathlibimport requestsPROXY = "http://USERNAME_pool-arcfast_country-de:PASSWORD@proxy.arcproxies.io:8000"CACHE = pathlib.Path("validators.json")validators = json.loads(CACHE.read_text()) if CACHE.exists() else {}def fetch(url: str): headers = {"Accept-Encoding": "gzip, br"} # br needs the brotli package installed v = validators.get(url, {}) if "etag" in v: headers["If-None-Match"] = v["etag"] if "last_modified" in v: headers["If-Modified-Since"] = v["last_modified"] r = requests.get(url, headers=headers, timeout=30, proxies={"http": PROXY, "https": PROXY}) if r.status_code == 304: return None # unchanged: no body transferred r.raise_for_status() validators[url] = {k: r.headers[h] for k, h in (("etag", "ETag"), ("last_modified", "Last-Modified")) if h in r.headers} CACHE.write_text(json.dumps(validators)) return r.textFAQ
Questions
Something not covered? Write to contact@arcproxies.io.
How can I reduce bandwidth when scraping?
Fetch HTML or the page's JSON instead of rendering in a browser; if you must render, block images, fonts and media; request compressed responses; use conditional requests so unchanged pages return 304 without a body; and cache and deduplicate so you never fetch the same thing twice.
Does blocking images break scraping?
Rarely. Most data is in the HTML or in JSON the page loads, not in images. A few pages wait for image or font events before showing content or lazy-load on scroll, so test extraction after you enable blocking.
What is a conditional request?
A request that includes a validator from a previous response — an ETag in If-None-Match or a date in If-Modified-Since. If the resource has not changed, the server answers 304 Not Modified with no body, so you transfer almost nothing.
Does gzip or Brotli compression reduce proxy bandwidth?
Yes, for text responses such as HTML and JSON. The client asks with an Accept-Encoding header and the server returns a compressed body. Most HTTP clients send the header by default; confirm yours does, and that it can decode Brotli if you request it.
Is it cheaper to scrape an API endpoint than the web page?
Almost always. A public JSON endpoint returns only the data, while a rendered page brings scripts, styles, fonts and images — about 2.6 MB on the median desktop page in 2024. Check the site's terms for any endpoint you use and keep the request rate polite.
Sources & review
Last reviewed by the ARC Proxies team. Prices are read from the live catalog at build time.
- [1]HTTP Archive: Web Almanac 2024 — Page Weight
- [2]IETF: RFC 9110: HTTP Semantics (conditional requests, content coding)
- [3]IETF: RFC 9111: HTTP Caching
- [4]IETF: RFC 7932: Brotli Compressed Data Format
- [5]Playwright: Network — Python
Next
Where to go from here
Guides