15% off your first purchase with code WELCOME15, up to 25 GB.

See pricing
ARC Proxies

Integration · Crawling frameworks

How to use a proxy with Colly

Colly sets a proxy with c.SetProxy("http://user:pass@host:port"), or a function per request with c.SetProxyFunc. Both switch off keep-alive on the collector's transport, so every request opens a new connection; with a rotating ARC username that can mean a new exit per request. For crawls that need a stable address, put a _session- handle in the username.

Reviewed 30 Oct 2026

your proxy, as a URL
http://USERNAME_pool-arcfast_country-de:PASSWORD@proxy.arcproxies.io:8000

The short answer

  • Build the proxy URL with url.UserPassword and pass its String() to SetProxy.
  • SetProxy disables keep-alive: each request is a new tunnel, and a rotating username can change exit every time.
  • proxy.RoundRobinProxySwitcher over several sticky usernames gives a fixed pool of exits.
  • Clone() shares the transport: calling SetProxy on a clone changes the parent's proxy too.

Setup

Set up Colly with a proxy, step by step

  1. 01

    Install Colly

    Colly v2 lives at github.com/gocolly/colly/v2.

    shell
    go get github.com/gocolly/colly/v2
  2. 02

    Set the proxy and check the exit IP

    SetProxy parses the URL and installs http.ProxyURL on the transport; net/http sends the userinfo as Basic proxy authentication.

    main.go
    package mainimport (	"fmt"	"log"	"net/url"	"time"	"github.com/gocolly/colly/v2")func arcProxy(username string) string {	u := &url.URL{Scheme: "http", User: url.UserPassword(username, "PASSWORD"), Host: "proxy.arcproxies.io:8000"}	return u.String()}func main() {	c := colly.NewCollector(colly.AllowURLRevisit())	c.SetRequestTimeout(30 * time.Second)	if err := c.SetProxy(arcProxy("USERNAME_pool-arcfast_country-de")); err != nil {		log.Fatal(err)	}	c.OnResponse(func(r *colly.Response) { fmt.Println(r.StatusCode, string(r.Body)) })	c.OnError(func(r *colly.Response, err error) { log.Println(r.StatusCode, err) })	c.Visit("https://api.ipify.org?format=json")	c.Visit("https://api.ipify.org?format=json") // new connection: may be a new exit}
  3. 03

    Spread a crawl over a fixed pool of sticky exits

    Give the round-robin switcher several sticky usernames. Requests cycle through them, so the crawl uses exactly that many exits, each held for _sesstime- minutes.

    pool.go
    package mainimport (	"fmt"	"log"	"github.com/gocolly/colly/v2"	"github.com/gocolly/colly/v2/proxy")func stickyPool(n int) []string {	urls := make([]string, n)	for i := range urls {		urls[i] = arcProxy(fmt.Sprintf("USERNAME_pool-arcfast_country-de_session-crawl%02d_sesstime-30", i))	}	return urls}func newPooledCollector() *colly.Collector {	c := colly.NewCollector()	rp, err := proxy.RoundRobinProxySwitcher(stickyPool(5)...)	if err != nil {		log.Fatal(err)	}	c.SetProxyFunc(rp)	return c}
  4. 04

    Keep one exit per site

    A custom ProxyFunc sees each *http.Request, so it can pick a username by host: every request to one site leaves from the same address, different sites from different ones.

    persite.go
    package mainimport (	"net/http"	"net/url"	"strings")func perSiteProxy(r *http.Request) (*url.URL, error) {	handle := strings.NewReplacer(".", "", "-", "").Replace(r.URL.Hostname())	user := "USERNAME_pool-arcfast_country-de_session-" + handle + "_sesstime-30"	return &url.URL{Scheme: "http", User: url.UserPassword(user, "PASSWORD"), Host: "proxy.arcproxies.io:8000"}, nil}// c.SetProxyFunc(perSiteProxy)
  5. 05

    Geo-target with one collector per market

    Use NewCollector for each market, not Clone(): clones share the parent's transport, so a proxy set on a clone replaces the parent's.

    inside main()
    markets := map[string]string{	"de-berlin":  "USERNAME_pool-arcfast_country-de_city-berlin",	"it-rome":    "USERNAME_pool-arcfast_country-it_city-rome",	"ca-toronto": "USERNAME_pool-arcfast_country-ca_city-toronto",}for market, user := range markets {	mc := colly.NewCollector()	mc.SetProxy(arcProxy(user))	mc.OnResponse(func(r *colly.Response) { fmt.Println(market, r.StatusCode) })	mc.OnError(func(r *colly.Response, err error) { fmt.Println(market, r.StatusCode, err) }) // 4xx/5xx land here	mc.Visit("https://example.com/price/123")}
  6. 06

    Handle 407, rate limits and retries

    A refused HTTPS tunnel reaches OnError with status 0 and "Proxy Authentication Required" in the error. Retry 429, 5xx and network errors with r.Request.Retry(), counting attempts in the request context, and slow down with a LimitRule.

    inside main(), before Visit
    c.Limit(&colly.LimitRule{DomainGlob: "*", Parallelism: 4, RandomDelay: 2 * time.Second})c.OnError(func(r *colly.Response, err error) {	if r.StatusCode == 407 || strings.Contains(err.Error(), "Proxy Authentication Required") {		log.Fatal("proxy rejected the credentials")	}	if r.StatusCode != 0 && r.StatusCode != 429 && r.StatusCode < 500 {		return // 404 and similar: do not retry	}	tries, _ := strconv.Atoi(r.Ctx.Get("tries"))	if tries >= 3 {		log.Println("giving up on", r.Request.URL)		return	}	r.Ctx.Put("tries", strconv.Itoa(tries+1))	time.Sleep(time.Duration(1<<tries) * time.Second)	r.Request.Retry()})

Rotation in Colly: why each request can get a new IP

Colly's SetProxyFunc, which SetProxy calls, sets DisableKeepAlives on the transport. Every request therefore opens a new CONNECT tunnel, and with a username that has no _session- token the gateway is free to pick a new exit each time. In our test two Visit calls produced two tunnels.

That suits wide crawls of independent pages. For paginated listings, search-then-detail flows or anything that keys on the visitor's address, use sticky usernames, either one per site or a small round-robin pool. Rotating vs sticky sessions goes through the choice.

Colly proxy options at a glance

MethodWhat it doesWatch out for
c.SetProxy(url)One proxy for every requestKeep-alive off
c.SetProxyFunc(fn)Your function picks the proxy per requestKeep-alive off; return a *url.URL
proxy.RoundRobinProxySwitcher(urls...)Cycles through the given proxiesSupports http, https and socks5 schemes
c.WithTransport(t)Replaces the transportCall it before SetProxy, or the proxy is lost with the old transport

Under the hood Colly is Go's net/http, so everything on the Go net/http page about Transport.Proxy applies. socks5:// URLs also work with the gateway's SOCKS5 listener; Go resolves DNS at the proxy for both socks5 and socks5h.

The problem

Common pitfalls

SetProxy on a Clone()

Clones share the parent's HTTP backend. Setting a proxy on the clone changed the parent's proxy in our test. Create a new collector per market or session.

Relying on Request.ProxyURL

With Colly 2.3.0 the field stayed empty in our test, both with SetProxy and with the round-robin switcher. Log the username you chose yourself.

Async without a LimitRule

colly.Async(true) with no limit sends as fast as Go can. Set Parallelism and a delay per domain; the 429 guide explains why.

Retrying a 407

Check for the proxy-auth error before calling Retry(); otherwise the retry loop repeats the same wrong password.

FAQ

Questions

Something not covered? Write to contact@arcproxies.io.

How do I set a proxy in Colly?

Call c.SetProxy("http://user:pass@host:port") on the collector. Build the URL with url.UserPassword so special characters in the password are escaped.

How do I rotate proxies in Colly?

With a rotating gateway you do not need a list: SetProxy turns off keep-alive, so each request opens a new connection and can get a new exit. To rotate across a fixed set of sticky sessions, pass several usernames to proxy.RoundRobinProxySwitcher.

Does Colly support SOCKS5 proxies?

Yes. Colly uses net/http, which accepts socks5://user:pass@host:port proxy URLs, and the round-robin switcher lists socks5 as a supported scheme.

Why does Colly's OnError show status 0 with a proxy?

Status 0 means no response came back from the site. With an HTTPS target and wrong proxy credentials, the gateway refused the CONNECT tunnel and the error reads "Proxy Authentication Required".

Sources & review

Last reviewed by the ARC Proxies team. Prices are read from the live catalog at build time.

  1. [1]Colly: colly package: SetProxy, SetProxyFunc, Clone (v2.3.0)
  2. [2]Colly: proxy package: RoundRobinProxySwitcher
  3. [3]GitHub: gocolly/colly colly.go (SetProxyFunc sets DisableKeepAlives)

Next

What people build with it

Use caseProxies for web scraping and data collectionA second provider that just works — Scrapy, requests, Crawlee or Playwright, self-serve, no sales call, bandwidth that never expires.Use caseProxies for price and competitor monitoringSee the price a shopper in Berlin, Lyon or Chicago sees. ISP bandwidth at our lowest price per GB for HTML checks, residential where a marketplace is strict.Use caseProxies for SEO and SERP monitoringCollect search results the way a searcher in that city sees them — unpersonalised, per country and language, at a price per GB that survives daily runs.GlossaryHTTP CONNECTHTTP CONNECT is the HTTP request method a client sends to a proxy to ask for a tunnel to a host and port; once the proxy answers with a 2xx status it blindly forwards bytes in both directions, which is how HTTPS passes through an HTTP proxy with TLS intact end to end (RFC 9110, section 9.3.6).GuideRotating vs sticky sessionsRotating proxies change exit IP per connection; sticky sessions hold one IP for a set time. How each works, when to use which, and how to set them up.GuideHandling 429 rate limitsWhat HTTP 429 means, how to read Retry-After, and how to retry politely with exponential backoff, jitter and per-host concurrency limits. With code.IntegrationGoRoute Go's net/http through an authenticated proxy: Transport.Proxy with url.UserPassword, a sticky session per request context, SOCKS5 and 407 checks.IntegrationScrapySet an authenticated proxy in Scrapy with meta['proxy'] and a small middleware: sticky sessions across pagination, geo per request, retries and SOCKS.

Integrations

Other crawling frameworks

CrawleeConfigure Crawlee's ProxyConfiguration with an authenticated gateway: rotating URLs, a sticky IP per session, geo per request, retries and concurrency.ScrapySet an authenticated proxy in Scrapy with meta['proxy'] and a small middleware: sticky sessions across pagination, geo per request, retries and SOCKS.scrapy-playwrightSet a proxy for scrapy-playwright: PLAYWRIGHT_LAUNCH_OPTIONS, a sticky session per browser context, markets as contexts and what a 407 looks like.

Get started

Get credentials and paste them in.

Sign up, buy a gigabyte, generate a username in the dashboard. Code WELCOME15 takes 15% off your first purchase; bandwidth never expires.

Create your accountQuestions first? contact@arcproxies.io