Glossary · Data collection
User-Agent
User-Agent is the HTTP request header in which a client identifies its software — one or more product tokens such as curl/8.5.0, with optional versions and comments — and which RFC 9110 says servers use to diagnose compatibility problems, tailor responses and compile browser and operating-system statistics.
Reviewed 30 Oct 2026
import requestsproxy = "http://USERNAME_pool-arcfast_country-de:PASSWORD@proxy.arcproxies.io:8000"headers = {"User-Agent": "ExamplePriceBot/1.2 (+https://example.com/bot)"}r = requests.get("https://example.com/", headers=headers, proxies={"http": proxy, "https": proxy}, timeout=30)What clients send by default
| Client | Default User-Agent shape |
|---|---|
| curl | curl/<version> |
| Python requests | python-requests/<version> |
| Chrome | Mozilla/5.0 (<platform>) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<major>.0.0.0 Safari/537.36 |
Browser strings start with Mozilla/5.0 for historical compatibility, which is exactly the kind of borrowed token RFC 9110 discourages in new software.
What RFC 9110 asks of a client
- Send a User-Agent on each request unless configured not to.
- Keep product identifiers to what identifies the product; no advertising in them.
- Avoid needlessly fine-grained detail: long, detailed values add latency and raise the risk of fingerprinting.
- Do not use other products' tokens to claim compatibility. If you masquerade as another agent, the server may assume you want responses tailored to that agent.
User-Agent Client Hints
Chrome now sends a reduced User-Agent string and moves detail into Client Hints headers. By default it sends Sec-CH-UA (brands and major version), Sec-CH-UA-Mobile and Sec-CH-UA-Platform; a server that wants more, such as the full version list, asks for it with an Accept-CH response header. A headless browser sends these on its own; a plain HTTP client sends none.
Choosing a User-Agent for collection
A proxy does not touch this header; the site sees exactly what your client sends. For a web crawler, a descriptive product token with a contact URL is the honest choice, and it is the token that robots.txt groups are matched against (RFC 9309). Setting it is a one-line option in Python requests and Playwright; the wider etiquette is in respecting robots.txt.
FAQ
Questions
Something not covered? Write to contact@arcproxies.io.
Does a proxy change my User-Agent?
No. Through an HTTPS tunnel the proxy cannot see or alter request headers at all, and a well-behaved proxy leaves them alone on plain HTTP too. The User-Agent is whatever your client sends.
Should a scraper send a browser User-Agent?
Only if the client really behaves like that browser. RFC 9110 discourages borrowing other products' tokens, and a server may then send markup your client cannot handle. For crawling, a descriptive token that robots.txt rules can address is the transparent option.
What are User-Agent Client Hints?
Headers such as Sec-CH-UA, Sec-CH-UA-Mobile and Sec-CH-UA-Platform that Chromium browsers send alongside a reduced User-Agent string. Servers request extra hints with the Accept-CH response header.
Can a website block requests by User-Agent?
Yes, and it is one of the simplest rules a site can apply, including in robots.txt groups for named crawlers. Respect such rules rather than changing the header to get past them.
Sources & review
Last reviewed by the ARC Proxies team. Prices are read from the live catalog at build time.
- [1]IETF: RFC 9110: HTTP Semantics
- [2]Chrome for Developers: Improving user privacy and developer experience with User-Agent Client Hints
- [3]IETF: RFC 9309: Robots Exclusion Protocol
In practice