Integration · Crawling frameworks

How to use a proxy with Crawlee

Q: How do I add a proxy with a username and password to Crawlee?

Put the credentials in the proxy URL: new ProxyConfiguration({ proxyUrls: ['http://user:pass@host:port'] }), then pass it as proxyConfiguration to the crawler. Crawlee uses it for HTTP crawlers and passes it on to the browsers it launches.

Q: How do I keep the same IP for a Crawlee session?

Use newUrlFunction(sessionId) and build the username from the session id, for example …_session- _sesstime-10. Every request in that session then uses the same ARC session handle and the same exit IP.

Q: How does Crawlee rotate proxies?

With proxyUrls, Crawlee picks a URL from the list per session. With a single ARC gateway URL, rotation happens at the gateway instead: a username without a _session- token can take a different exit on each new connection.

Q: Can I use a different country per request in Crawlee?

Yes. newUrlFunction receives { request } as its second argument, so you can read a country from request.userData and put it in the _country- token of the username.

Crawlee takes proxies through one object, ProxyConfiguration, and every crawler (Cheerio, Playwright, Puppeteer, HTTP) accepts it. Give it the ARC gateway as a URL with credentials and it rotates; give it a newUrlFunction that turns Crawlee's session id into an ARC _session- handle and each Crawlee session keeps its own exit IP.

Reviewed 30 Oct 2026

your proxy, as a URL1 line

http://USERNAME_pool-arcfast_country-de:PASSWORD@proxy.arcproxies.io:8000

The short answer

proxyUrls: ['http://user:pass@host:port'] is enough to start; the gateway does the rotating.
newUrlFunction(sessionId) maps each Crawlee session to an ARC session handle, so the session pool and the exit IP move together.
When a session is retired after a 401, 403 or 429, the next session gets a new handle and therefore a new exit.
The same ProxyConfiguration works for CheerioCrawler, PlaywrightCrawler and PuppeteerCrawler; Crawlee handles browser proxy credentials for you.

Setup

Set up Crawlee with a proxy, step by step

01
Install Crawlee
This guide uses Crawlee 3 for Node. Add playwright only if you use PlaywrightCrawler.
shell2 lines
```
npm i crawleenpm i playwright   # only for PlaywrightCrawler
```

Rotating proxy in one line, and check the exit

One proxy URL with the rotating username: each new connection can leave from a different exit. proxyInfo in the handler tells you which proxy a request used, which is the quickest way to confirm the config is live. The IP echo response confirms the exit itself.

rotating.mjs15 lines

import { CheerioCrawler, ProxyConfiguration } from 'crawlee';const proxyConfiguration = new ProxyConfiguration({  proxyUrls: ['http://USERNAME_pool-arcfast_country-de:PASSWORD@proxy.arcproxies.io:8000'],});const crawler = new CheerioCrawler({  proxyConfiguration,  additionalMimeTypes: ['application/json'],   // let the echo response through  async requestHandler({ request, json, proxyInfo }) {    console.log(request.url, json?.ip ?? '', proxyInfo?.hostname);  },});await crawler.run(['https://api.ipify.org?format=json']);

A sticky exit IP per Crawlee session

newUrlFunction receives Crawlee's session id. Strip it to letters and digits and use it as the ARC handle: a session keeps its IP for _sesstime- minutes, and a retired session takes its IP with it.

sticky.mjs21 lines

import { CheerioCrawler, ProxyConfiguration } from 'crawlee';const proxyConfiguration = new ProxyConfiguration({  newUrlFunction: (sessionId) => {    // Crawlee ids look like "session_Ab12Cd34Ef"; keep [a-z0-9] for the handle.    const handle = String(sessionId).replace(/[^a-z0-9]/gi, '').toLowerCase();    return `http://USERNAME_pool-arcfast_country-de_session-${handle}_sesstime-10:PASSWORD@proxy.arcproxies.io:8000`;  },});const crawler = new CheerioCrawler({  proxyConfiguration,  useSessionPool: true,              // the default, shown for clarity  persistCookiesPerSession: true,    // cookies and IP stay together  maxConcurrency: 10,  async requestHandler({ $, request, session }) {    console.log(session?.id, request.url, $('title').text());  },});await crawler.run(['https://example.com/category/1', 'https://example.com/category/2']);

Choose the country per request

newUrlFunction also receives the request, so a country stored in userData can pick the username. Country codes are lowercase ISO; each country here still gets a sticky handle per session.

geo.mjs12 lines

const proxyConfiguration = new ProxyConfiguration({  newUrlFunction: (sessionId, { request } = {}) => {    const country = request?.userData?.country ?? 'de';    const handle = String(sessionId).replace(/[^a-z0-9]/gi, '').toLowerCase();    return `http://USERNAME_pool-arcfast_country-${country}_session-${handle}_sesstime-10:PASSWORD@proxy.arcproxies.io:8000`;  },});await crawler.run([  { url: 'https://example.com/fr/prices', userData: { country: 'fr' } },  { url: 'https://example.com/it/prices', userData: { country: 'it' } },]);

Use the same config with PlaywrightCrawler

Browser crawlers take the same object. Crawlee passes the proxy, credentials included, to the browser it launches, so there is no page.authenticate() to call.

browser.mjs13 lines

import { PlaywrightCrawler } from 'crawlee';const crawler = new PlaywrightCrawler({  proxyConfiguration,                 // from the sticky example above  maxConcurrency: 4,                  // browsers are heavier than HTTP requests  maxRequestRetries: 3,  async requestHandler({ page, request }) {    console.log(request.url, await page.title());  },  failedRequestHandler({ request }, error) {    console.error('gave up', request.url, error.message);  },});

How Crawlee's session pool and ARC sessions fit together

Crawlee's session pool tracks a set of sessions, each with its own cookies and error score. With newUrlFunction mapping session id to _session- handle, each Crawlee session is also one ARC exit. Three defaults in Crawlee 3.18 matter:

Setting	Default	What it means with ARC
Blocked status codes	401, 403, 429	The session is retired; its replacement mints a new handle, so the retry leaves from a new IP.
`maxRequestRetries`	3	Ordinary failures (timeouts, 5xx) are retried up to three times.
`maxSessionRotations`	10	Session rotations after blocks are counted separately from `maxRequestRetries`.

That is the sticky session model with replacement built in. Sticky holds are best effort, so a session's IP can still change before _sesstime- ends; Crawlee treats the resulting error like any other and retries.

Concurrency and rate limits

maxConcurrency caps parallel requests across the crawler; maxRequestsPerMinute caps the rate. Start low per target domain and raise it while watching the 429 rate. More parallelism through more IPs does not change what a site considers reasonable, and its robots.txt and terms still apply. Handling 429 rate limits and the web scraping use case cover the pacing side.

Crawlee for Python and Crawlee 4

Crawlee for Python has the same concept: ProxyConfiguration(proxy_urls=['http://user:pass@host:port']) from crawlee.proxy_configuration, passed to the crawler's proxy_configuration.
Crawlee 4 is a release candidate on npm at the time of writing and changes parts of the proxy API. The snippets here target Crawlee 3, the current latest release.

The problem

Common pitfalls

Session ids used raw as handles

Crawlee's ids contain an underscore (session_…), and underscores separate tokens in the ARC username. Strip the id to letters and digits first, as in the examples.

Logging proxyInfo.url

proxyInfo.url contains the password. Log proxyInfo.hostname or the session id instead.

Browser concurrency set like HTTP concurrency

A maxConcurrency that is fine for CheerioCrawler can exhaust memory with PlaywrightCrawler, and every browser page also costs far more bandwidth than its HTML.

FAQ

Questions

Something not covered? Write to contact@arcproxies.io.

How do I add a proxy with a username and password to Crawlee?

Put the credentials in the proxy URL: new ProxyConfiguration({ proxyUrls: ['http://user:pass@host:port'] }), then pass it as proxyConfiguration to the crawler. Crawlee uses it for HTTP crawlers and passes it on to the browsers it launches.

How do I keep the same IP for a Crawlee session?

Use newUrlFunction(sessionId) and build the username from the session id, for example …_session-<id>_sesstime-10. Every request in that session then uses the same ARC session handle and the same exit IP.

How does Crawlee rotate proxies?

With proxyUrls, Crawlee picks a URL from the list per session. With a single ARC gateway URL, rotation happens at the gateway instead: a username without a _session- token can take a different exit on each new connection.

Can I use a different country per request in Crawlee?

Yes. newUrlFunction receives { request } as its second argument, so you can read a country from request.userData and put it in the _country- token of the username.

Sources & review

Last reviewed 30 October 2026 by the ARC Proxies team. Prices are read from the live catalog at build time.

[1]Crawlee: Proxy management (JavaScript)
[2]Crawlee: ProxyConfiguration API reference
[3]Crawlee: Proxy management (Python)

What people build with it

Use caseProxies for web scraping and data collectionA second provider that just works — Scrapy, requests, Crawlee or Playwright, self-serve, no sales call, bandwidth that never expires.Use caseProxies for AI and LLM training-data collectionCollect public web text, images and product data at corpus scale from residential and ISP IPs, paid per gigabyte with no monthly commitment.Use caseProxies for price and competitor monitoringSee the price a shopper in Berlin, Lyon or Chicago sees. ISP bandwidth at our lowest price per GB for HTML checks, residential where a marketplace is strict.GlossarySticky sessionA sticky session is a setting on a rotating proxy that keeps the same exit IP address for a chosen period, usually by attaching a session identifier to the connection, so a multi-step task appears to come from one address from start to finish.GlossaryRotating proxyA rotating proxy is a proxy endpoint that automatically assigns a different exit IP address from a pool to new connections or requests, so one client's traffic is spread across many addresses without the client managing a list.GuideHandling 429 rate limitsWhat HTTP 429 means, how to read Retry-After, and how to retry politely with exponential backoff, jitter and per-host concurrency limits. With code.IntegrationPlaywrightSet up an authenticated proxy in Playwright for Python and Node: launch and per-context proxies, sticky sessions, geo-targeting and handling proxy errors.Use caseProxies for web scraping and data collectionA second provider that just works — Scrapy, requests, Crawlee or Playwright, self-serve, no sales call, bandwidth that never expires.

Integrations

Other crawling frameworks

ScrapySet an authenticated proxy in Scrapy with meta['proxy'] and a small middleware: sticky sessions across pagination, geo per request, retries and SOCKS.CollyUse an authenticated proxy with the Go crawler Colly: SetProxy, a pool of sticky sessions with RoundRobinProxySwitcher, per-site exits and retries.scrapy-playwrightSet a proxy for scrapy-playwright: PLAYWRIGHT_LAUNCH_OPTIONS, a sticky session per browser context, markets as contexts and what a 407 looks like.

Get started

Get credentials and paste them in.

Sign up, buy a gigabyte, generate a username in the dashboard. Code WELCOME15 takes 15% off your first purchase; bandwidth never expires.

Create your account Questions first? contact@arcproxies.io