15% off your first purchase with code WELCOME15, up to 25 GB.

See pricing
ARC Proxies

Integration · Crawling frameworks

How to use a proxy with Scrapy

Scrapy proxies a request when request.meta["proxy"] is set; the built-in HttpProxyMiddleware reads it, moves the credentials from the URL into a Proxy-Authorization header, and the download handler opens the connection. A ten-line middleware that sets meta["proxy"] on every request is all an ARC setup needs.

Reviewed 30 Oct 2026

your proxy, as a URL
http://USERNAME_pool-arcfast_country-de:PASSWORD@proxy.arcproxies.io:8000

The short answer

  • meta["proxy"] = "http://user:pass@host:port" per request; it takes precedence over http_proxy environment variables.
  • A custom middleware with priority below 750 runs before HttpProxyMiddleware and can choose the username per request.
  • Carry a session handle in meta through pagination to keep one exit for one listing.
  • A 407 surfaces as TunnelError, which is in Scrapy's default retry list, so wrong credentials are retried twice before failing.

Setup

Set up Scrapy with a proxy, step by step

  1. 01

    Start from courteous settings

    A new Scrapy project's template already obeys robots.txt and paces requests per domain. Keep that, and add AutoThrottle so the delay follows the site's response times. Handling 429 rate limits explains why pacing matters more than the number of exits.

    settings.py
    ROBOTSTXT_OBEY = TrueCONCURRENT_REQUESTS_PER_DOMAIN = 1   # raise slowly, per targetDOWNLOAD_DELAY = 1AUTOTHROTTLE_ENABLED = TrueDOWNLOADER_MIDDLEWARES = {    "myproject.middlewares.ArcProxyMiddleware": 740,   # before HttpProxyMiddleware (750)}
  2. 02

    Add a middleware that sets the proxy

    Every request gets the rotating username unless it carries an arc_session handle, in which case it gets that sticky session. Note the signature: Scrapy 2.19 warns when a middleware method requires the spider argument.

    middlewares.py
    BASE = "USERNAME_pool-arcfast_country-de"GATEWAY = "proxy.arcproxies.io:8000"PASSWORD = "PASSWORD"class ArcProxyMiddleware:    def process_request(self, request, spider=None):        user = BASE        if handle := request.meta.get("arc_session"):            user += f"_session-{handle}_sesstime-10"        request.meta["proxy"] = f"http://{user}:{PASSWORD}@{GATEWAY}"        return None
  3. 03

    Check the exit, then keep one IP across a listing

    The first request hits an IP echo service to confirm the setup. Each listing then starts with a fresh handle and passes it to its next page, so the whole walk through one listing uses one exit, while product pages rotate.

    spiders/products.py
    import secretsimport scrapyclass ProductsSpider(scrapy.Spider):    name = "products"    async def start(self):        yield scrapy.Request("https://api.ipify.org?format=json", callback=self.check_exit)    def check_exit(self, response):        self.logger.info("exit IP: %s", response.json()["ip"])        for url in ["https://example.com/c/lamps", "https://example.com/c/chairs"]:            yield scrapy.Request(url, callback=self.parse_listing,                                 meta={"arc_session": secrets.token_hex(4)})    def parse_listing(self, response):        for href in response.css("a.product::attr(href)").getall():            yield response.follow(href, self.parse_product)          # rotating        if next_page := response.css("a[rel=next]::attr(href)").get():            yield response.follow(next_page, self.parse_listing,      # same exit                                  meta={"arc_session": response.meta["arc_session"]})    def parse_product(self, response):        yield {"url": response.url, "title": response.css("h1::text").get()}
  4. 04

    Choose the country per request

    Extend the middleware to read a country from meta. Country codes are lowercase ISO; add _city- for a metro from the published list.

    middlewares.py
    class ArcGeoProxyMiddleware:    def process_request(self, request, spider=None):        country = request.meta.get("arc_country", "de")        user = f"USERNAME_pool-arcfast_country-{country}"        if handle := request.meta.get("arc_session"):            user += f"_session-{handle}_sesstime-10"        request.meta["proxy"] = f"http://{user}:PASSWORD@proxy.arcproxies.io:8000"        return None# yield scrapy.Request(url, meta={"arc_country": "fr"})
  5. 05

    Separate proxy failures from site failures

    An errback sees failures after retries are exhausted. A TunnelError mentioning 407 is a credentials problem, so raising CloseSpider stops the crawl; other tunnel errors and timeouts are worth logging and re-queuing later. 429 and 5xx are retried by RetryMiddleware already.

    spiders/products.py
    import scrapyfrom scrapy.core.downloader.handlers.http11 import TunnelErrorfrom scrapy.exceptions import CloseSpider, DownloadTimeoutErrorclass ProductsSpider(scrapy.Spider):    # ... start() and callbacks as above, with errback=self.on_error on each Request    def on_error(self, failure):        if failure.check(TunnelError) and "407" in str(failure.value):            raise CloseSpider("proxy_auth_failed")       # wrong credentials: stop        if failure.check(TunnelError, DownloadTimeoutError):            self.logger.warning("proxy or timeout failure on %s", failure.request.url)

Retries, AutoThrottle and what Scrapy retries by default

Scrapy's defaults already cover most of the retry logic a proxy setup needs. In Scrapy 2.19:

SettingDefaultWith ARC
RETRY_TIMES2Three attempts in total. With a rotating username, each retry can leave from a new exit.
RETRY_HTTP_CODES500, 502, 503, 504, 522, 524, 408, 429429 is retried; AutoThrottle lengthens the delay when responses slow down.
RETRY_EXCEPTIONSIncludes timeouts and TunnelErrorA 407 tunnel is retried too, which only delays the failure.
DOWNLOAD_TIMEOUT180 sLower it (for example 30 s) so a stalled exit is replaced sooner.

For pages that need several requests from one visitor, carry the arc_session handle as in the listing example. A retry keeps the handle; if the exit itself went away, the sticky session gets a new IP on the next connection anyway.

How Scrapy handles proxy credentials and connections

  • HttpProxyMiddleware strips user:pass from meta["proxy"] and sends it as Proxy-Authorization. When meta["proxy"] changes on a request, it replaces or removes the old header, so a middleware that changes the username per retry is safe.
  • The HTTP/1.1 download handler keys pooled tunnels by proxy host, port and credentials. Requests with different usernames never share a tunnel; requests with the same rotating username to the same host can, and then share an exit.
  • Use an http:// proxy URL for HTTPS sites. Scrapy's default handler supports HTTPS proxies only for HTTP destinations; HTTPS sites go through an HTTP CONNECT tunnel.

SOCKS5 in Scrapy

The default download handlers do not speak SOCKS. Scrapy 2.15 added an experimental HttpxDownloadHandler that supports SOCKS5 proxy URLs in meta["proxy"]; it needs the httpx extra and an asyncio reactor, and it opens a separate connection pool per proxy URL, which adds up when every request has its own handle. For most crawls, the HTTP gateway does the same job with fewer moving parts, and how much bandwidth scraping uses matters more to the bill than the protocol.

The problem

Common pitfalls

Middleware priority above 750

A middleware that sets meta["proxy"] after HttpProxyMiddleware has run leaves the credentials in the URL and no header. Use a number below 750.

Turning robots.txt off to go faster

The template sets ROBOTSTXT_OBEY = True for good reason. More IPs are not permission; keep it on and stay within each site's terms.

Using an https:// proxy URL

The gateway is reached over plain HTTP. https:// in meta["proxy"] breaks HTTPS targets with the default handler.

Concurrency raised before measuring

CONCURRENT_REQUESTS_PER_DOMAIN multiplies load on one site regardless of how many exits you have. Raise it step by step and watch the 429 count.

FAQ

Questions

Something not covered? Write to contact@arcproxies.io.

How do I set a proxy for a Scrapy request?

Set request.meta["proxy"] = "http://user:pass@host:port", either when you create the request or in a downloader middleware with priority below 750. HttpProxyMiddleware turns the credentials into a Proxy-Authorization header.

How do I rotate proxies in Scrapy?

Point every request at a rotating gateway username, and each new connection can take a new exit, including retries. For explicit control, give each request or each listing its own _session- handle in the username.

Why does Scrapy say Could not open CONNECT tunnel with proxy?

The proxy refused the CONNECT request for an HTTPS URL. If the message shows status 407, the credentials are wrong or missing; other statuses or timeouts point to a transient proxy or network error that a retry may fix.

Can Scrapy use a SOCKS5 proxy?

Not with the default handlers. Scrapy 2.15+ includes an experimental HttpxDownloadHandler that accepts SOCKS5 proxy URLs. An HTTP proxy URL works with the default handlers and reaches the same gateway.

Does meta['proxy'] override http_proxy environment variables?

Yes. Scrapy's docs state that a proxy meta value takes precedence over http_proxy and https_proxy, and no_proxy is ignored for that request.

Sources & review

Last reviewed by the ARC Proxies team. Prices are read from the live catalog at build time.

  1. [1]Scrapy: Downloader middleware: HttpProxyMiddleware
  2. [2]Scrapy: Downloader middleware: RetryMiddleware
  3. [3]Scrapy: Download handlers

Next

What people build with it

Use caseProxies for web scraping and data collectionA second provider that just works — Scrapy, requests, Crawlee or Playwright, self-serve, no sales call, bandwidth that never expires.Use caseProxies for price and competitor monitoringSee the price a shopper in Berlin, Lyon or Chicago sees. ISP bandwidth at our lowest price per GB for HTML checks, residential where a marketplace is strict.Use caseProxies for real-estate and job-market dataListings and postings are ranked by where the visitor is. Collect them city by city so your index reflects the market, not your server's location.Use caseProxies for web scraping and data collectionA second provider that just works — Scrapy, requests, Crawlee or Playwright, self-serve, no sales call, bandwidth that never expires.GlossarySticky sessionA sticky session is a setting on a rotating proxy that keeps the same exit IP address for a chosen period, usually by attaching a session identifier to the connection, so a multi-step task appears to come from one address from start to finish.GlossaryHTTP CONNECTHTTP CONNECT is the HTTP request method a client sends to a proxy to ask for a tunnel to a host and port; once the proxy answers with a 2xx status it blindly forwards bytes in both directions, which is how HTTPS passes through an HTTP proxy with TLS intact end to end (RFC 9110, section 9.3.6).GuideHandling 429 rate limitsWhat HTTP 429 means, how to read Retry-After, and how to retry politely with exponential backoff, jitter and per-host concurrency limits. With code.GuideHow much bandwidth scraping usesBandwidth per page for HTML-only and browser-rendered scraping, using HTTP Archive page-weight data, with a worked table for 100,000 pages.

Integrations

Other crawling frameworks

CrawleeConfigure Crawlee's ProxyConfiguration with an authenticated gateway: rotating URLs, a sticky IP per session, geo per request, retries and concurrency.CollyUse an authenticated proxy with the Go crawler Colly: SetProxy, a pool of sticky sessions with RoundRobinProxySwitcher, per-site exits and retries.scrapy-playwrightSet a proxy for scrapy-playwright: PLAYWRIGHT_LAUNCH_OPTIONS, a sticky session per browser context, markets as contexts and what a 407 looks like.

Get started

Get credentials and paste them in.

Sign up, buy a gigabyte, generate a username in the dashboard. Code WELCOME15 takes 15% off your first purchase; bandwidth never expires.

Create your accountQuestions first? contact@arcproxies.io