15% off your first purchase with code WELCOME15, up to 25 GB.

See pricing
ARC Proxies

Integration · AI & workflow tools

How to use a proxy with LangChain document loaders

LangChain's web loaders accept a proxies dictionary, but it only reaches the synchronous requests code path. WebBaseLoader(proxies=…).load() goes through the proxy; its async methods and AsyncHtmlLoader.load() fetch with aiohttp and, in our test, went out directly. The setting that covers every path is the HTTPS_PROXY environment variable plus trust_env=True; for AsyncHtmlLoader alone, requests_kwargs={"proxy": url} also works.

Reviewed 30 Oct 2026

your proxy, as a URL
http://USERNAME_pool-arcfast_country-de:PASSWORD@proxy.arcproxies.io:8000

The short answer

  • WebBaseLoader(proxies={...}) proxies load() and lazy_load(), not aload() or alazy_load().
  • AsyncHtmlLoader(proxies={...}) stores the proxy but its load() never uses it. Pass requests_kwargs={"proxy": url} instead.
  • HTTPS_PROXY in the environment with trust_env=True on the loader proxied every path we tested, sync and async.
  • These loaders live in langchain-community, whose repository was archived on 19 June 2026. Pin the version you tested.

Setup

Set up LangChain loaders with a proxy, step by step

  1. 01

    Install the loaders

    WebBaseLoader parses with BeautifulSoup; AsyncHtmlLoader returns raw HTML. Set a USER_AGENT environment variable, or LangChain logs a warning on import.

    shell
    pip install "langchain-community==0.4.2" beautifulsoup4export USER_AGENT="research-bot/1.0 (+https://example.com/bot)"
  2. 02

    WebBaseLoader with proxies (synchronous load)

    The dictionary uses the requests format: one entry per target scheme, both pointing at the gateway with http://. HTTPS pages still work: requests opens a CONNECT tunnel through the proxy.

    web_base.py
    from langchain_community.document_loaders import WebBaseLoaderPROXY = "http://USERNAME_pool-arcfast_country-de_session-k3f9a1x2_sesstime-10:PASSWORD@proxy.arcproxies.io:8000"loader = WebBaseLoader(    ["https://api.ipify.org?format=json", "https://example.com/"],    proxies={"http": PROXY, "https": PROXY},)docs = loader.load()                # synchronous: goes through the proxyprint(docs[0].page_content)         # {"ip":"…"}: the exit, not your machine
  3. 03

    AsyncHtmlLoader: pass the proxy to aiohttp

    AsyncHtmlLoader.load() runs on aiohttp, which takes a single proxy URL per request. requests_kwargs is passed straight into each session.get() call, so that is where the proxy goes.

    async_html.py
    from langchain_community.document_loaders import AsyncHtmlLoaderPROXY = "http://USERNAME_pool-arcfast_country-de:PASSWORD@proxy.arcproxies.io:8000"loader = AsyncHtmlLoader(    ["https://example.com/a", "https://example.com/b"],    requests_kwargs={"proxy": PROXY},    requests_per_second=2,        # concurrent requests, despite the name)docs = loader.load()
  4. 04

    One setting for every path: environment plus trust_env

    With HTTPS_PROXY set and trust_env=True, aiohttp reads the proxy from the environment, and requests reads it by default. This is the option to use in a RAG pipeline that mixes load() and alazy_load().

    env_proxy.py
    import osfrom langchain_community.document_loaders import WebBaseLoaderos.environ["HTTPS_PROXY"] = "http://USERNAME_pool-arcfast_country-de:PASSWORD@proxy.arcproxies.io:8000"os.environ["HTTP_PROXY"] = os.environ["HTTPS_PROXY"]loader = WebBaseLoader(["https://example.com/a", "https://example.com/b"], trust_env=True)docs = loader.load()                               # requests: proxied# async for doc in loader.alazy_load(): ...       # aiohttp: proxied via trust_env
  5. 05

    Handle proxy errors without logging the password

    A rejected password raises requests.exceptions.ProxyError on the sync path and aiohttp.ClientHttpProxyError (status 407) on the async path. The aiohttp message includes the proxy URL with the password in it, so log the status, not the exception text.

    errors.py
    import aiohttpimport requestsdef load_safely(loader):    try:        return loader.load()    except requests.exceptions.ProxyError:        raise RuntimeError("proxy refused the connection: check username and password") from None    except aiohttp.ClientHttpProxyError as err:        raise RuntimeError(f"proxy returned {err.status}") from None   # no URL, no password

Which LangChain loader path uses the proxy

Measured with langchain-community 0.4.2 against a local proxy that logs every connection:

Callproxies={...}requests_kwargs={"proxy": url}HTTPS_PROXY + trust_env=True
WebBaseLoader.load() / lazy_load()ProxiedBreaks: requests rejects the keywordProxied
WebBaseLoader.alazy_load() / aload()Not proxiedProxiedProxied
AsyncHtmlLoader.load()Not proxiedProxiedProxied

"Not proxied" fails silently: the page loads from your own IP and nothing in the result tells you. Run the IP echo check through the exact method your pipeline calls.

Rotating or sticky for RAG ingestion

Loading a list of unrelated documentation or article URLs is a natural fit for a rotating username: each connection can leave from a different exit, which spreads the load. Use a sticky username, with _session- and _sesstime-, when the URLs are steps of one browse on one site, such as paginated listings that set cookies. Rotating vs sticky sessions has the details.

The loaders do not render JavaScript. For pages that need a browser, a rendering crawler such as Crawl4AI can produce the documents instead, and its markdown output feeds a text splitter the same way.

Polite ingestion with LangChain loaders

  • Keep requests_per_second low; it caps concurrency, and two to five parallel requests per site is plenty for most ingestion jobs.
  • Check a site's robots.txt and terms before bulk loading, and stick to public pages. Building AI training data covers collection hygiene.
  • Cache what you load. Re-fetching an unchanged page costs bandwidth on a per-GB plan and adds load to the site.

The problem

Common pitfalls

Trusting proxies= on the async path

The most common failure here is invisible: async methods ignore the proxies dictionary and fetch directly. Use the environment variable with trust_env=True, or test each method you call.

Using requests_kwargs on WebBaseLoader's sync path

WebBaseLoader passes requests_kwargs to both requests and aiohttp. A proxy key works for aiohttp but makes requests.get() fail, so do not use it with load().

Passwords in logs

aiohttp includes the proxy URL, credentials and all, in its error message. Catch the exception and log only its status.

An unpinned, archived package

langchain-community no longer receives maintenance. Pin the version, and re-run the proxy check whenever you upgrade anything in the stack.

FAQ

Questions

Something not covered? Write to contact@arcproxies.io.

How do I use a proxy with LangChain WebBaseLoader?

Pass proxies={"http": url, "https": url} for load(). For async methods, set HTTPS_PROXY in the environment and create the loader with trust_env=True; that combination proxied every path in our test.

Why does AsyncHtmlLoader ignore my proxies argument?

AsyncHtmlLoader.load() fetches with aiohttp, and the proxies dictionary is only applied to an internal requests session that load() does not use. Pass requests_kwargs={"proxy": url} or use HTTPS_PROXY with trust_env=True.

Is langchain-community still maintained?

LangChain announced the sunset of langchain-community in May 2026 and the repository was archived on 19 June 2026. The package still installs; pin the version and plan for a replacement.

Can LangChain loaders use a SOCKS5 proxy?

Not without extra packages: requests needs requests[socks] and aiohttp needs a connector such as aiohttp-socks. The ARC gateway serves HTTP on the same port with the same credentials, so the http:// form needs nothing extra.

Sources & review

Last reviewed by the ARC Proxies team. Prices are read from the live catalog at build time.

  1. [1]LangChain: WebBaseLoader reference: proxies, requests_kwargs, trust_env
  2. [2]LangChain (GitHub): Sunsetting langchain-community (issue #674)
  3. [3]aiohttp: Client advanced usage: proxy support and trust_env

Next

What people build with it

Use caseProxies for AI and LLM training-data collectionCollect public web text, images and product data at corpus scale from residential and ISP IPs, paid per gigabyte with no monthly commitment.Use caseResidential proxies for AI agentsBring your own residential IPs to browser-use, Playwright, Crawl4AI, n8n and any MCP tool — one sticky session per task, billed per GB.Use caseProxies for web scraping and data collectionA second provider that just works — Scrapy, requests, Crawlee or Playwright, self-serve, no sales call, bandwidth that never expires.GlossaryHTTP CONNECTHTTP CONNECT is the HTTP request method a client sends to a proxy to ask for a tunnel to a host and port; once the proxy answers with a 2xx status it blindly forwards bytes in both directions, which is how HTTPS passes through an HTTP proxy with TLS intact end to end (RFC 9110, section 9.3.6).GuideRotating vs sticky sessionsRotating proxies change exit IP per connection; sticky sessions hold one IP for a set time. How each works, when to use which, and how to set them up.IntegrationPython requestsUse an authenticated proxy with Python requests: the proxies argument, sticky sessions per task, geo-targeting, SOCKS5 and retries that skip 407 errors.IntegrationaiohttpUse an authenticated proxy with aiohttp: proxy= per request or per session, why proxy_auth is deprecated, SOCKS5 through aiohttp-socks, and 407 handling.IntegrationCrawl4AIUse an authenticated proxy in Crawl4AI: ProxyConfig on CrawlerRunConfig, a sticky session per crawl task, locale-matched geo-targeting and error checks.Use caseProxies for AI and LLM training-data collectionCollect public web text, images and product data at corpus scale from residential and ISP IPs, paid per gigabyte with no monthly commitment.

Integrations

Other ai & workflow tools

n8nRoute n8n HTTP Request nodes through an authenticated proxy: the Proxy option, sticky sessions per execution, instance-wide env vars and 407 errors.Crawl4AIUse an authenticated proxy in Crawl4AI: ProxyConfig on CrawlerRunConfig, a sticky session per crawl task, locale-matched geo-targeting and error checks.Firecrawl (self-hosted)Route a self-hosted Firecrawl through an authenticated proxy with PROXY_SERVER, PROXY_USERNAME and PROXY_PASSWORD, then test it with a /v2/scrape call.StagehandStagehand's local browser rejects proxy usernames and passwords. Route it through an authenticated proxy with a local forwarder, sticky per agent task.

Get started

Get credentials and paste them in.

Sign up, buy a gigabyte, generate a username in the dashboard. Code WELCOME15 takes 15% off your first purchase; bandwidth never expires.

Create your accountQuestions first? contact@arcproxies.io