Guide · Scraping practice
How to handle HTTP 429 Too Many Requests
HTTP 429 Too Many Requests means the server is rate-limiting you: you sent more requests than it allows in a period. Handle it by stopping, waiting at least as long as the Retry-After header says, retrying with exponential backoff plus random jitter, and lowering your concurrency for that host. A 429 is a polite request to slow down; treating it as an obstacle to route around is what turns a rate limit into a block.
Reviewed 30 Oct 2026
The short answer
- 429 is defined in RFC 6585. The server may include Retry-After; the spec does not define how the server identifies you or counts requests.
- Retry-After is either a number of seconds or an HTTP date (RFC 9110). Honour it when present.
- Without Retry-After, use capped exponential backoff with full jitter — random wait between zero and the current cap — so clients do not retry in lockstep.
- Limit concurrency per host and slow the whole host down after a 429, not just the one request.
- Well-behaved crawlers do this already: Google reduces its crawl rate across a hostname when it sees many 429, 500 or 503 responses.
What a 429 response means
RFC 6585 defines 429 as the user having sent too many requests in a given amount of time. The response should explain the condition and may include a Retry-After header saying how long to wait.
The RFC is deliberately open about how the limit works: a server may count per resource, across the whole server or across a set of servers, and may identify the client by credentials or a stateful cookie. So do not assume the limit is per IP or per URL. Assume it applies to you, and slow down. See rate limiting.
Status codes and what to do
| Status | Meaning | Action |
|---|---|---|
| 429 Too Many Requests | You exceeded the server's rate limit | Wait Retry-After (or back off), then retry; reduce concurrency for the host |
| 503 Service Unavailable | Server overloaded or in maintenance | Same as 429; Retry-After says how long it expects to be unavailable |
| 500, 502, 504 | Server or upstream error | Back off and retry a limited number of times |
| 403 Forbidden | Access refused | Do not retry in a loop; check robots.txt, terms and whether the content is public |
| 404, 410 | Not there | Do not retry; record and move on |
Reading Retry-After
RFC 9110 defines two forms: a number of seconds (Retry-After: 120 means two minutes) or an HTTP date (Retry-After: Fri, 31 Dec 1999 23:59:59 GMT). Parse both. If the value is in the past or unparseable, fall back to your backoff schedule.
Treat Retry-After as a minimum, not a target. If many workers hit the same host, waking them all at exactly the stated second recreates the burst that caused the 429 — add jitter on top.
Exponential backoff with jitter
Exponential backoff doubles the wait after each failure up to a cap. On its own it still lets many clients retry at the same moments. AWS's analysis of retry strategies found that adding jitter dramatically reduces the total work clients do, and recommends “full jitter”: wait a random time between zero and the current backoff cap.
In practice: sleep = random(0, min(cap, base × 2^attempt)), with a base of about one second, a cap of a minute or two, and a maximum number of attempts after which the URL goes back into the queue for a later run.
Concurrency per host, and courtesy
- Cap concurrent requests per host, not just globally. A global limit of 50 can still put all 50 on one site.
- When a host returns 429, reduce that host's concurrency (for example halve it) and restore it slowly after a period without errors.
- Spread work over time. A schedule that fires everything on the hour creates its own rate limit.
- Read robots.txt and any published crawl guidance — see collecting travel data responsibly. Google's documentation describes its own crawlers reducing crawl rate across a whole hostname when they see a significant number of 429, 500 or 503 responses — a good model for any collector.
- Spreading requests across rotating proxy exits changes how per-IP counting sees you; it does not add capacity to the site. Use it to avoid concentrating a legitimate workload on one address, not to exceed what the site is willing to serve. See our acceptable use policy.
Code
Worked example
import asyncio, randomfrom email.utils import parsedate_to_datetimefrom datetime import datetime, timezonefrom urllib.parse import urlsplitimport httpxPROXY = "http://USERNAME_pool-arcfast_country-de:PASSWORD@proxy.arcproxies.io:8000"PER_HOST = 4 # concurrent requests per hostBASE, CAP, MAX_ATTEMPTS = 1.0, 90.0, 6host_slots: dict[str, asyncio.Semaphore] = {}def retry_after_seconds(value: str | None) -> float | None: if not value: return None if value.strip().isdigit(): return float(value) try: when = parsedate_to_datetime(value) return max(0.0, (when - datetime.now(timezone.utc)).total_seconds()) except (TypeError, ValueError): return Noneasync def fetch(client: httpx.AsyncClient, url: str) -> httpx.Response | None: slot = host_slots.setdefault(urlsplit(url).hostname, asyncio.Semaphore(PER_HOST)) for attempt in range(MAX_ATTEMPTS): async with slot: r = await client.get(url) if r.status_code not in (429, 500, 502, 503, 504): return r backoff = random.uniform(0, min(CAP, BASE * 2 ** attempt)) # full jitter wait = max(retry_after_seconds(r.headers.get("Retry-After")) or 0, backoff) await asyncio.sleep(wait + random.uniform(0, 1)) return None # give up for now; re-queue for a later runasync def main(urls): async with httpx.AsyncClient(proxy=PROXY, timeout=30) as client: return await asyncio.gather(*(fetch(client, u) for u in urls))asyncio.run(main(["https://example.com/page/1", "https://example.com/page/2"]))FAQ
Questions
Something not covered? Write to contact@arcproxies.io.
What does HTTP 429 Too Many Requests mean?
It means the server is rate-limiting you: you sent more requests than it allows in a given time. It is defined in RFC 6585, and the response may include a Retry-After header saying how long to wait before trying again.
How long should I wait after a 429?
At least as long as the Retry-After header says, if it is present — either a number of seconds or an HTTP date. If it is absent, use exponential backoff with random jitter, starting around one second and capped at a minute or two.
What is exponential backoff with jitter?
A retry strategy where the maximum wait doubles after each failure and the actual wait is a random time between zero and that maximum. The randomness stops many clients retrying at the same instant; AWS's analysis found it greatly reduces total load compared with plain backoff.
Should I rotate IPs when I get a 429?
Not as a way around the limit. A 429 is the server asking you to slow down, and RFC 6585 notes it may identify you by cookie or credentials, not just IP. Slow down for that host first. Rotation is for spreading a polite workload, not for exceeding what a site will serve.
What is the difference between 429 and 503?
429 says you specifically sent too many requests; 503 says the service is unavailable, often overloaded or in maintenance. Both can carry Retry-After, and a polite client backs off on both.
How many retries should a scraper make?
A handful — five or six attempts with capped exponential backoff is common — after which the URL goes back into the queue for a later run. Unlimited retries turn a rate limit into a flood and waste bandwidth.
Sources & review
Last reviewed by the ARC Proxies team. Prices are read from the live catalog at build time.
- [1]IETF: RFC 6585: Additional HTTP Status Codes (429 Too Many Requests)
- [2]IETF: RFC 9110: HTTP Semantics (Retry-After)
- [3]AWS Architecture Blog: Exponential Backoff And Jitter
- [4]Google Search Central: Reduce the Googlebot crawl rate
- [5]IETF: RFC 9309: Robots Exclusion Protocol
Next
Where to go from here
Guides