Glossary · Data collection
Rate limiting
Rate limiting is a server's cap on how many requests a client may make in a given time; when the cap is exceeded the standard response is HTTP 429 Too Many Requests (RFC 6585), often with a Retry-After header saying how long to wait.
Reviewed 30 Oct 2026
import random, timeimport requestsproxy = "http://USERNAME_pool-arcfast_country-de:PASSWORD@proxy.arcproxies.io:8000"proxies = {"http": proxy, "https": proxy}def get(url, attempts=5): delay = 2.0 for _ in range(attempts): r = requests.get(url, proxies=proxies, timeout=30) if r.status_code not in (429, 503): return r wait = r.headers.get("Retry-After", "") time.sleep(int(wait) if wait.isdigit() else delay + random.random()) delay *= 2 return rHow servers count requests
RFC 6585 deliberately does not say how a server identifies a client or counts its requests. In practice limits are applied per IP address, per cookie or session, per account, per endpoint, or across a whole site, and expensive endpoints such as search are usually limited hardest.
429 is not the only signal. A 503 with Retry-After, a sudden run of empty results with status 200, or an interstitial challenge page all mean the same thing: slow down.
Responding to 429 correctly
- Honour
Retry-After. RFC 9110 allows either a number of seconds or an HTTP date. - Without it, back off exponentially with jitter rather than retrying at a fixed interval.
- Cap concurrency per host, not just overall, and cache anything that has not changed.
- Do not answer a 429 by replaying the same burst through fresh addresses straight away.
Patterns and code for each client are in handling 429 rate limits.
Where proxies fit, and where they do not
A proxy pool stops one address from carrying a whole collection job, which is what makes a reasonable crawl possible at all. It does not change what the site can serve. A limit is the operator telling you its capacity; stay within it, follow robots.txt and the site's terms, and keep collection to public pages. The ground rules are on the acceptable use page.
FAQ
Questions
Something not covered? Write to contact@arcproxies.io.
What does HTTP 429 Too Many Requests mean?
The client has sent more requests in a given time than the server allows. RFC 6585 defines the status, says the response should explain the condition, and allows a Retry-After header telling the client when to try again. 429 responses must not be cached.
What is the Retry-After header?
A response header that tells the client how long to wait before the next request, as either a number of seconds or an HTTP date. RFC 9110 defines it; servers send it with 429, 503 and some redirects.
Do rate-limited requests use proxy bandwidth?
Yes, the bytes they transfer. On ARC billing is per gigabyte of traffic, so a 429 costs the small response it returned and nothing more, but a retry storm adds up. Backing off is cheaper as well as politer.
Will rotating proxies stop me from being rate limited?
They spread per-address counters, nothing else. Limits keyed on cookies, sessions or request patterns still apply, and a site's limit is its stated capacity. Plan the job to stay within it.
Sources & review
Last reviewed by the ARC Proxies team. Prices are read from the live catalog at build time.
- [1]IETF: RFC 6585: Additional HTTP Status Codes
- [2]IETF: RFC 9110: HTTP Semantics
In practice