Guide · Proxy basics
How to calculate proxy costs for a scraping project
On per-GB billing, proxy cost is pages per month × average bytes transferred per page × an overhead factor for retries and protocol bytes, multiplied by the price per gigabyte of the pool you use. The middle term varies most: an HTML or JSON fetch is measured in kilobytes, a fully rendered page in megabytes. Measure bytes per page on your own targets before you budget, and the rest is arithmetic.
Reviewed 30 Oct 2026
The short answer
- GB per month = pages per month × KB per page × (1 + retry rate) × (1 + protocol overhead) ÷ 1,000,000, in decimal units.
- Measure KB per page as bytes on the wire — compressed body plus headers — on 20–50 real pages, and use the median and the 90th percentile.
- Failed requests, redirects and challenge pages pass through the proxy too. Budget them.
- Choose the pool per job: plain HTML monitors and rendered browser sessions rarely belong on the same pool.
- Multiply GB by the per-GB price for each pool on the pricing page, then check cost per 1,000 successful pages.
The formula, term by term
- Pages per month. Items × refreshes per day × 30. A 5,000-product catalogue checked daily is 150,000 pages.
- KB per page. Bytes transferred for one successful page, measured on the wire. HTTP Archive's 2024 Web Almanac puts the median HTML document at 18 KB and the median full desktop page at 2,652 KB; your targets will differ, which is why you measure.
- Retry rate. Extra requests as a share of pages — 0.10 if one page in ten needs a second attempt.
- Protocol overhead. Request and response headers, redirects and TLS handshakes on new connections. Small per request, large across millions of small ones; 5% is a starting assumption to replace with a measurement.
- Price per GB. From the pricing page, for the pool and volume tier you will buy.
Worked scenarios
Each row uses the formula with a 10% retry rate and 5% protocol overhead. The per-page sizes are assumptions for illustration — replace them with your measurements.
| Scenario | Pages per month | Assumed per page | GB per month |
|---|---|---|---|
| 5,000 product pages daily, HTML only | 150,000 | 60 KB | ≈ 10.4 |
| 20,000 SKUs hourly from a JSON endpoint | 14,400,000 | 8 KB | ≈ 133.1 |
| 500 keywords × 3 countries daily, result pages | 45,000 | 90 KB | ≈ 4.7 |
| 10 hotels × 540 stay combinations × 2 markets daily, JSON | 324,000 | 45 KB | ≈ 16.8 |
| 1,000 pages daily rendered in a browser, images blocked | 30,000 | 1,600 KB | ≈ 55.4 |
The hourly JSON row shows that frequency, not page weight, can dominate: a light response requested 24 times a day outweighs a heavy one requested once. The rendered row shows the opposite: 30,000 pages cost more than five times the 150,000 HTML pages.
Measuring bytes per page
- Ask for compression with
Accept-Encodingand measure the body as it arrives, before decompression — that is what crosses the proxy. In Python requests, readr.rawwithdecode_content=False;len(r.content)is the decompressed size and overstates it. - Add response headers, and request headers if your cookies are large.
- For browsers, sum transferred bytes across all responses in the page's network log, after your blocking rules apply.
- Sample 20–50 pages per target, including the heavy ones, and keep the median and the 90th percentile. Budget on the 90th if a shortfall would stop the job.
- After a pilot, compare your estimate with the usage your provider reports. That figure is the one you are billed on.
Overheads people forget
- Retries and failures. Every failed attempt, error page and challenge page is traffic.
- Redirects. Two requests for one page if the first answers with a redirect.
- New connections. A TLS handshake per new connection; reusing connections with a session avoids repeating it for every request.
- Headers. HTTP/2 compresses header fields with HPACK (RFC 7541), HTTP/1.1 does not; on small JSON responses headers can be a noticeable share.
- Third-party assets in browsers. Trackers, fonts and ads load unless blocked.
- robots.txt and discovery. Sitemaps, category pages and robots.txt fetches are pages too.
Each of these can be reduced — see how to reduce proxy bandwidth. For page-weight data by resource type, see how much bandwidth scraping uses.
From gigabytes to a budget
- Split the workload by pool: HTML monitors and API polling often run on a rotating ISP pool such as ISP proxies, browser sessions and stricter targets on residential proxies.
- Multiply each pool's GB by its per-GB price at the tier you will buy; see per-GB pricing for how volume tiers work.
- Add a margin for growth and for re-runs after a parser bug.
- Track cost per 1,000 successful pages over time. It is the number that tells you whether a pool change, a retry policy or a page redesign made things better or worse.
Because our bandwidth does not expire, buying at a higher volume tier ahead of need does not waste the remainder; on expiring plans, budget per month instead.
Code
Worked example
import statisticsimport requestsPROXY = "http://USERNAME_pool-arcfast_country-de:PASSWORD@proxy.arcproxies.io:8000"def wire_bytes(session: requests.Session, url: str) -> int: """Bytes of one response as transferred: body as sent (still compressed) plus headers.""" r = session.get(url, proxies={"http": PROXY, "https": PROXY}, headers={"Accept-Encoding": "gzip, deflate"}, timeout=(5, 30), stream=True) body = r.raw.read(decode_content=False) # do not decompress headers = sum(len(k) + len(v) + 4 for k, v in r.headers.items()) return len(body) + headersdef sample(urls: list[str]) -> dict: with requests.Session() as s: sizes = sorted(wire_bytes(s, u) for u in urls) return {"p50_kb": statistics.median(sizes) / 1000, "p90_kb": sizes[int(0.9 * (len(sizes) - 1))] / 1000}def monthly_gb(pages_per_month: int, kb_per_page: float, retry_rate: float = 0.10, overhead: float = 0.05) -> float: """GB = pages x KB x (1 + retries) x (1 + protocol overhead) / 1,000,000 (decimal units).""" return pages_per_month * kb_per_page * (1 + retry_rate) * (1 + overhead) / 1_000_000# 5,000 product pages a day, HTML only, measured p50 of 60 KB:print(round(monthly_gb(5_000 * 30, 60), 1), "GB per month") # -> 10.4FAQ
Questions
Something not covered? Write to contact@arcproxies.io.
How many GB do I need for web scraping?
Multiply pages per month by the bytes per page you measure on your targets, add 10–20% for retries and protocol overhead, and divide by 1,000,000 KB per GB. 150,000 HTML pages at 60 KB come to roughly 10 GB a month; the same number of rendered pages can be over a hundred times more.
How do I estimate proxy bandwidth before I start?
Fetch 20–50 representative pages per target with the method you will use in production, measure the bytes on the wire, and take the median and 90th percentile. Multiply by planned volume and add an overhead margin.
Do failed requests count towards proxy bandwidth?
Yes. Any traffic that passes through the proxy counts, including error pages, redirects and retries, so a retry rate belongs in every estimate.
Is per-GB or per-request pricing cheaper for scraping?
Per-GB pricing favours small responses such as HTML and JSON; per-request pricing favours heavy pages. Compute both with your measured bytes per page and compare cost per successful page.
Does compression reduce proxy costs?
Usually, yes. Proxy traffic is counted in bytes transferred, and a compressed HTML or JSON response is often several times smaller than the decompressed one. Make sure your client sends Accept-Encoding.
Are headers counted in proxy bandwidth?
Yes. Request and response headers cross the proxy like the body does. They matter most for small responses and large cookies; HTTP/2 compresses them, HTTP/1.1 does not.
Sources & review
Last reviewed by the ARC Proxies team. Prices are read from the live catalog at build time.
- [1]HTTP Archive: Web Almanac 2024 — Page Weight
- [2]IETF: RFC 7541: HPACK — Header Compression for HTTP/2
- [3]Requests: Advanced Usage: Body Content Workflow and Response.raw
Next
Where to go from here
Guides