Integration · Crawling frameworks

How to use a proxy with scrapy-playwright

Q: How do I set a proxy in scrapy-playwright?

Add "proxy": {"server": "http://host:port", "username": "user", "password": "pass"} to PLAYWRIGHT_LAUNCH_OPTIONS for the whole browser, or to a context in PLAYWRIGHT_CONTEXTS or playwright_context_kwargs for one context.

Q: How do I use a different proxy per request in scrapy-playwright?

Give the request its own context: set playwright_context to a new name and pass the proxy in playwright_context_kwargs. Every request that names that context then shares its proxy.

scrapy-playwright does not use Scrapy's proxy meta key for pages it renders: the browser makes those requests, so the proxy goes to the browser. Set it once in PLAYWRIGHT_LAUNCH_OPTIONS = {"proxy": {"server": ..., "username": ..., "password": ...}}, or per browser context through PLAYWRIGHT_CONTEXTS or the playwright_context_kwargs meta key. A context is the natural unit for a sticky session or a market.

Reviewed 30 Oct 2026

your proxy, as a URL1 line

http://USERNAME_pool-arcfast_country-de:PASSWORD@proxy.arcproxies.io:8000

The short answer

Browser-wide proxy: PLAYWRIGHT_LAUNCH_OPTIONS["proxy"] with server, username, password.
Per context: playwright_context_kwargs={"proxy": {...}}. One context per sticky session, closed when the task ends.
meta["proxy"] is ignored on requests with "playwright": True; it still applies to plain Scrapy requests in the same spider.
Wrong proxy credentials showed up as a navigation timeout, not a 407 response.

Setup

Set up scrapy-playwright with a proxy, step by step

01
Install the plugin and a browser
Playwright is installed as a dependency; the browsers are not.
shell2 lines
```
pip install scrapy-playwrightplaywright install chromium
```

Activate the handler and set a browser-wide proxy

Register the Playwright download handler, use the asyncio reactor, and give the browser the gateway. Every request marked "playwright": True in the default context then leaves through it.

settings.py15 lines

DOWNLOAD_HANDLERS = {    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",}TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"PLAYWRIGHT_LAUNCH_OPTIONS = {    "proxy": {        "server": "http://proxy.arcproxies.io:8000",        "username": "USERNAME_pool-arcfast_country-de",        "password": "PASSWORD",    },}PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT = 30_000   # msPLAYWRIGHT_MAX_CONTEXTS = 8

Check the exit IP from the browser

A minimal spider renders an IP echo page through the gateway.

exit_ip.py14 lines

import scrapyclass ExitIpSpider(scrapy.Spider):    name = "exit_ip"    async def start(self):        yield scrapy.Request("https://api.ipify.org?format=json", meta={"playwright": True})    def parse(self, response):        # Chromium wraps the JSON in a small HTML page; the IP is in the text        yield {"status": response.status, "body": response.text[:300]}# run inside the project that has the settings above:  scrapy crawl exit_ip -O exit.jsonl

One sticky session per browser context

Name a new context per task and pass its proxy in playwright_context_kwargs. Requests that name the same context share its cookies and its exit. Close the page and the context when the task is done, also on errors, or PLAYWRIGHT_MAX_CONTEXTS blocks the crawl.

listing_spider.py42 lines

import secretsimport scrapyGATEWAY = "http://proxy.arcproxies.io:8000"class ListingSpider(scrapy.Spider):    name = "listing"    async def start(self):        for start_url in ["https://example.com/c/lamps", "https://example.com/c/chairs"]:            handle = secrets.token_hex(4)            yield scrapy.Request(                start_url,                meta={                    "playwright": True,                    "playwright_include_page": True,                    "playwright_context": f"session-{handle}",                    "playwright_context_kwargs": {                        "proxy": {                            "server": GATEWAY,                            "username": f"USERNAME_pool-arcfast_country-de_session-{handle}_sesstime-10",                            "password": "PASSWORD",                        },                    },                },                callback=self.parse_listing,                errback=self.close_context,            )    async def parse_listing(self, response):        page = response.meta["playwright_page"]        for href in response.css("a.product::attr(href)").getall():            yield {"url": response.urljoin(href)}        await page.close()        await page.context.close()    async def close_context(self, failure):        page = failure.request.meta.get("playwright_page")        if page:            await page.close()            await page.context.close()

Define markets as named contexts

For a fixed set of countries or cities, declare the contexts in settings and pick one per request with playwright_context.

settings.py10 lines

PLAYWRIGHT_CONTEXTS = {    "default": {"proxy": {"server": "http://proxy.arcproxies.io:8000", "username": "USERNAME_pool-arcfast_country-de", "password": "PASSWORD"}},    "gb-london": {"proxy": {"server": "http://proxy.arcproxies.io:8000",                            "username": "USERNAME_pool-arcfast_country-gb_city-london", "password": "PASSWORD"}},    "us-seattle": {"proxy": {"server": "http://proxy.arcproxies.io:8000",                             "username": "USERNAME_pool-arcfast_country-us_city-seattle", "password": "PASSWORD"}},}# in the spider:# yield scrapy.Request(url, meta={"playwright": True, "playwright_context": "gb-london"})

06
Save bandwidth by aborting heavy resources
Every image, font and video the browser loads goes through the gateway and is billed per GB. PLAYWRIGHT_ABORT_REQUEST drops them before they are sent.
settings.py5 lines
```
def should_abort_request(request):    return request.resource_type in {"image", "media", "font"}PLAYWRIGHT_ABORT_REQUEST = should_abort_request
```

Which proxy setting applies to which request

Request	Proxy comes from	Notes
`meta={"playwright": True}`, default context	`PLAYWRIGHT_LAUNCH_OPTIONS["proxy"]` or `PLAYWRIGHT_CONTEXTS["default"]`	Rendered by the browser
`playwright_context="name"`	That context's `proxy`	Kwargs apply only when the context is created
No `playwright` key	Scrapy's `meta["proxy"]` / `HttpProxyMiddleware`	Handled by the regular Scrapy download handler
`meta["proxy"]` on a playwright request	Ignored	The browser's proxy was used in our test

Mixing both is useful: render the listing page in the browser and fetch JSON endpoints or detail pages with plain Scrapy requests, which are cheaper. The Scrapy proxy guide covers the meta["proxy"] side, and the Playwright guide the browser options themselves.

How a wrong password shows up

Chromium answers the gateway's 407 challenge with the context's credentials. When they are wrong, Playwright does not hand back a 407 page: in our test the navigation waited until PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT and the errback received a TimeoutError. If every request in one context times out while others work, check that context's username first. The proxy authentication glossary entry explains the challenge itself.

Rotating or sticky with a browser

A browser opens several connections per page and keeps them alive, so a rotating username can give one page load more than one exit. Sites that tie a session to the visitor's address then see it jump. For rendered crawling, one sticky session per context is the safer default; use rotating usernames for independent single-page fetches. Rotating vs sticky sessions goes deeper.

The problem

Common pitfalls

Setting meta["proxy"] and expecting the browser to use it

Playwright-rendered requests ignore it. Put the proxy in the launch options or the context.

Changing context kwargs for an existing context

If a context with that name already exists, playwright_context_kwargs is ignored. Use a new name per session.

Contexts that are never closed

With PLAYWRIGHT_MAX_CONTEXTS set, unclosed contexts eventually stall the crawl. Close page and context in the callback and in an errback.

SOCKS5 with credentials

Playwright's Chromium refuses a SOCKS5 proxy with a username ("Browser does not support socks5 proxy authentication" in our test). Use the gateway's HTTP protocol in the server URL.

FAQ

Questions

Something not covered? Write to contact@arcproxies.io.

How do I set a proxy in scrapy-playwright?

Add "proxy": {"server": "http://host:port", "username": "user", "password": "pass"} to PLAYWRIGHT_LAUNCH_OPTIONS for the whole browser, or to a context in PLAYWRIGHT_CONTEXTS or playwright_context_kwargs for one context.

Does scrapy-playwright use Scrapy's proxy meta key?

Not for requests rendered by Playwright; the browser uses its own proxy setting. Requests without the playwright meta key still go through Scrapy's normal handler and do use meta["proxy"].

How do I use a different proxy per request in scrapy-playwright?

Give the request its own context: set playwright_context to a new name and pass the proxy in playwright_context_kwargs. Every request that names that context then shares its proxy.

Why do my scrapy-playwright requests time out behind a proxy?

A rejected proxy login can look like a navigation timeout, because the browser never gets past the 407 challenge. Check the username and password of that context, then the timeout and site speed.

Sources & review

Last reviewed 30 October 2026 by the ARC Proxies team. Prices are read from the live catalog at build time.

[1]scrapy-playwright: Proxy support
[2]scrapy-playwright: Browser contexts
[3]scrapy-playwright: Settings reference
[4]Playwright: Network: HTTP Proxy

What people build with it

Use caseProxies for web scraping and data collectionA second provider that just works — Scrapy, requests, Crawlee or Playwright, self-serve, no sales call, bandwidth that never expires.Use caseProxies for price and competitor monitoringSee the price a shopper in Berlin, Lyon or Chicago sees. ISP bandwidth at our lowest price per GB for HTML checks, residential where a marketplace is strict.Use caseProxies for travel fare aggregationAirline, hotel and rail fares change with the point of sale. Query each market from an IP inside it so the fare you store is the fare a traveller is quoted.IntegrationScrapySet an authenticated proxy in Scrapy with meta['proxy'] and a small middleware: sticky sessions across pagination, geo per request, retries and SOCKS.IntegrationPlaywrightSet up an authenticated proxy in Playwright for Python and Node: launch and per-context proxies, sticky sessions, geo-targeting and handling proxy errors.GlossaryProxy authenticationProxy authentication is how a proxy confirms that a client may use it: under RFC 9110 the proxy answers with status 407 Proxy Authentication Required and a Proxy-Authenticate challenge, and the client sends its credentials in a Proxy-Authorization header.GlossaryHeadless browserA headless browser is a real web browser engine running without a visible window and controlled by code instead of a person; it loads pages, executes JavaScript and builds the DOM like a normal browser, which makes it the tool for pages whose content only appears after scripts run.GuideRotating vs sticky sessionsRotating proxies change exit IP per connection; sticky sessions hold one IP for a set time. How each works, when to use which, and how to set them up.

Integrations

Other crawling frameworks

CrawleeConfigure Crawlee's ProxyConfiguration with an authenticated gateway: rotating URLs, a sticky IP per session, geo per request, retries and concurrency.ScrapySet an authenticated proxy in Scrapy with meta['proxy'] and a small middleware: sticky sessions across pagination, geo per request, retries and SOCKS.CollyUse an authenticated proxy with the Go crawler Colly: SetProxy, a pool of sticky sessions with RoundRobinProxySwitcher, per-site exits and retries.

Get started

Get credentials and paste them in.

Sign up, buy a gigabyte, generate a username in the dashboard. Code WELCOME15 takes 15% off your first purchase; bandwidth never expires.

Create your account Questions first? contact@arcproxies.io