Market research · Social listening
Social listening data from public forums, news and communities
Much of what a market says about a brand, a product or an issue is written in public forums, news comment sections and community sites in its own language. Collect those logged-out pages from inside each market, on a schedule you can repeat, and your sentiment series measures the conversation rather than your crawler's location.
Pay per GB · No subscription · Prices in EUR, excl. VAT
- €0.75
- per GB on ARC Basic, excl. VAT
- −21%
- at 1 TB (€0.59 / GB)
- 185+
- countries, city targeting
- Never
- expires — no subscription
Written for
- Consumer insights teams
- Social listening analysts
- Computational social science researchers
- Communications and issues managers
The short answer
- Social listening at scale is mostly public forums, news sites, community Q&A and review pages — content a logged-out visitor can read. Anything behind a login is out of scope.
- For large platforms, start with official APIs and research-access programmes. In the EU, non-commercial researchers can request publicly accessible data from very large platforms under the Digital Services Act.
- Collect each market's sources from an exit in that market, so regional editions, language defaults and geo-restricted pages match the audience you are measuring.
- Forum and news pages are light HTML; ARC Basic carries most of the volume, with residential for stricter sources.
The problem
What goes wrong without the right IPs
The sample is your crawler's country
News sites and portals serve regional editions and default languages by visitor location. Crawl a pan-European listening panel from one place and half of it quietly becomes one country's edition.
The big platforms are mostly behind a login
Many social networks show little or nothing to a logged-out visitor. Trying to listen there with a crawler is both unreliable and outside what we support; their APIs and research programmes are the right route.
Threads change after you read them
Replies arrive for days, posts are edited and deleted. A thread collected once at publication understates the conversation and misses the moment it turned.
Small forums cannot take a burst
A hobby forum on a single server is not built for a crawler fetching hundreds of pages a minute. Too fast and you degrade the community you are trying to listen to.
Personal data arrives with the text
Usernames, signatures and quoted posts come with every page. What you keep, for how long and on what basis is a decision to make before the crawl, not after.
What to collect
What you can collect
- Source, section and URL
- Which forum, board or news section — the frame for every sentiment number.
- Market and language
- Detected language plus the market the source serves; sentiment models are language-specific.
- Post or comment text
- The unit of analysis, stored with its position in the thread.
- Published and edited timestamps
- Lets you rebuild how a conversation developed and catch edits.
- Thread structure
- Reply-to relationships; a reply to a complaint is different from a new complaint.
- Engagement counts shown
- Replies, views or votes where the page shows them — a proxy for reach.
- Pseudonymised author key
- Count distinct voices without storing names; hash or drop the handle per your policy.
- Collection timestamp and revision
- Which pass saw which version of the page.
How it runs
How a collection run works
- 01
Build a source list per market
List the forums, news sites, comment sections and community sites that matter in each market and language. Check each one's robots.txt and terms, and drop sources that exclude crawlers or require a login.
- 02
Discover new threads from index pages
Poll board and section index pages on a schedule and queue threads that are new or have new replies. This is where most of the request count goes, and it is cheap: index pages are small.
- 03
Fetch threads from the market's own country
Rotating exits with the country token set for each source's market. Each page is an independent, logged-out request; paging a long thread in one sticky session is only needed where the site keeps pagination state.
Scrapy: per-source country, polite by default14 lines# settings.pyROBOTSTXT_OBEY = TrueAUTOTHROTTLE_ENABLED = TrueDOWNLOAD_DELAY = 1.0CONCURRENT_REQUESTS_PER_DOMAIN = 2# in the spiderGATEWAY = "proxy.arcproxies.io:8000"PASSWORD = "PASSWORD"def proxy_for(country: str) -> str: return f"http://USERNAME_pool-arc_country-{country}:{PASSWORD}@{GATEWAY}"# yield scrapy.Request(url, meta={"proxy": proxy_for(source.country)}) - 04
Revisit threads while they are alive
Re-fetch active threads on a decaying schedule — after an hour, a day, a week — and stop when no new replies arrive. That captures the conversation without re-crawling dead threads forever.
- 05
Minimise before you store
Drop or hash usernames, strip signatures and contact details, and keep only what the analysis needs. Retention limits belong in the pipeline, not in a policy document.
Setup
Recommended setup
Listening is many small, light pages across many sources, so the cheapest pool carries most of it, and residential handles the stricter sources.
| Plan | Role | Session | Targeting | Price |
|---|---|---|---|---|
ARC Basic Rotating ISP | Default for forums and news Rotating ISP exits per source market at our lowest price per GB. Forum and news pages are light HTML and suit it well. | Rotating | Country | €0.75 / GB €0.59 at 1 TB (−21%) |
ARC Fast Rotating residential | For stricter public sources Residential exits for the public sources where ISP addresses see failures you cannot otherwise explain, and for regional editions keyed to a metro. | Rotating, or sticky per thread | Country + city | €2.00 / GB €1.12 at 1 TB (−44%) |
Prices are list prices in EUR, excl. VAT. Volume discounts apply automatically at checkout — full ladder on the pricing page.
Connect
One connection string per market
The country, city and sticky session are tokens in the username. Change the token, change the market; keep the session handle, keep the IP.
http://USERNAME_pool-arcfast_country-de_session-k3f9a1x2_sesstime-10:PASSWORD@proxy.arcproxies.io:8000Swap _country-de for any market you quote. The full grammar is on the residential proxies page.
Illustration
What a five-market listening panel costs
An illustration with round numbers — your pages, payloads and retry rate will differ. The arithmetic is the real price ladder.
Assumptions
- Markets
- 5
- Pages fetched per day, all sources
- 30,000
- Mix
- index pages, new threads, revisits
- Weight per page, images blocked
- 120 KB
- Bandwidth per month
- ≈ 108 GB
On ARC Basic
€73.44per month, excl. VAT
108 GB bought in one purchase at €0.68 / GB (−9% volume rung, applied automatically).
The same 108 GB on ARC Fast: €146.88 at €1.36 / GB. Route only the jobs that need it through the pricier pool.
Image-heavy communities and news pages with large scripts weigh more; block media and you stay close to this. A paused project keeps its balance, because bandwidth never expires.
Bandwidth never expires, so a bigger single purchase is only ever cheaper per GB. See the ladder.
In depth
Social listening, in depth
Where public conversation actually lives
Social media matters for listening because audiences increasingly get their news and opinions there. The Reuters Institute's Digital News Report 2025 found that 54% of people in the United States accessed news via social media and video networks, overtaking TV (50%) and news websites and apps (48%) for the first time; across the markets it tracks, the proportion consuming social video news grew from 52% in 2020 to 65% in 2025.
But much of that conversation is not on pages a logged-out visitor can read. What is public, at scale and per market, is the long tail: national forums, enthusiast communities, Q&A sites, news comment sections, product review pages, and blogs. For most consumer categories that is where the detailed, in-language discussion lives — the material sentiment models are actually good at.
| Source type | Logged-out access | Best route |
|---|---|---|
| National and enthusiast forums | Usually public | Crawl index pages, then threads; ARC Basic per market |
| News sites and comment sections | Articles public; comments vary | Crawl articles per regional edition; comments only where public |
| Q&A and community sites | Often public | Crawl, respecting robots.txt; some offer data dumps or APIs |
| Large social networks | Often limited without login | Official APIs and research-access programmes — not a crawler |
Research access under the Digital Services Act
If you are an academic or a non-profit researcher studying the EU, the Digital Services Act gives you a route that is better than any crawler. Under Article 40(12), providers of designated very large online platforms and search engines must give researchers access to publicly accessible data without undue delay, and in real time where technically possible.
The Commission's FAQ lists the conditions: researchers must be independent from commercial interests, disclose the funding of their research, be able to meet data security and confidentiality requirements, and request only data that is proportionate and necessary. Commercial insights teams do not qualify — for them, the platforms' commercial APIs and the public long tail described above are the realistic sources.
Why the exit market matters for listening
A listening panel is a sample, and the sample is only as good as its frame. Regional news editions, language defaults and geo-restricted pages all follow the visitor's location, so the country you crawl from becomes part of the sampling design whether you choose it or not.
Setting the country per source makes it a choice. The Italian panel is crawled from Italian exits, the Polish panel from Polish ones, and when a regional edition is keyed to a metro — Milan versus Rome, Warsaw versus Krakow — the city token narrows it further. The parent market research page covers the wider sampling question; geo-targeting explains the tokens.
Collecting from communities without harming them
- Respect robots.txt. The Robots Exclusion Protocol is standardised as RFC 9309; read it per source and honour it.
- Keep concurrency per domain low. Two parallel requests and a second's pause is a good default for small forums; let auto-throttling slow you further.
- Identify yourself where you can. A user-agent with a contact address lets a forum administrator reach you instead of blocking you.
- Stop on a 429. A rate-limit response is the site's capacity talking — see handling 429 rate limits.
- Never log in, never get past a CAPTCHA. Public, logged-out pages only; our acceptable use policy is explicit about it.
Personal data in public posts
Public is not the same as anonymous. Under the GDPR, personal data is any information relating to an identified or identifiable natural person — and a username with a posting history can be exactly that. Decide before the crawl what you keep, why, and for how long, and take the specifics to your data protection officer or counsel.
In practice most listening analysis needs the text, the time, the source and a count of distinct voices — not the names. Hashing author handles at ingestion and dropping signatures and quoted contact details keeps the dataset useful and much smaller. The same discipline applies to multilingual corpora built from similar sources.
Works with anything that takes a proxy URL
- Scrapy
- Python requests / httpx
Crawlee- LangChain loaders
- Firecrawl (self-hosted)
- Docker
FAQ
Questions
Something not covered? Write to contact@arcproxies.io.
Can I use proxies for social listening?
Yes, for public, logged-out sources: forums, news sites and comment sections, Q&A and community sites, and review pages, collected per market from exits in that market. For large social networks, use their official APIs or research-access programmes; anything behind a login is outside our acceptable use policy.
Why collect social listening data from different countries?
Because news sites and portals serve regional editions and default languages by visitor location, and some pages are geo-restricted. Crawling every market from one place silently biases the sample towards that place. Setting the country per source makes the sampling frame a deliberate choice.
Can researchers get social media data under the Digital Services Act?
Researchers who are independent from commercial interests, disclose their funding and can meet data-protection and security requirements can request publicly accessible data from very large online platforms and search engines under Article 40(12) of the DSA. Providers must give that access without undue delay.
Is public forum data personal data?
It can be. The GDPR defines personal data as any information relating to an identified or identifiable person, and a username with a posting history may qualify. Minimise at collection — hash handles, drop signatures — and check your basis and retention with your data protection officer or counsel.
How much bandwidth does social listening use?
Forum and news pages are light once media is blocked, around 100–150 KB each. A five-market panel fetching 30,000 pages a day at 120 KB is about 108 GB a month. Bandwidth is billed per GB and never expires.
Which proxy plan suits forum and news crawling?
ARC Basic: rotating ISP exits with country targeting at our lowest price per GB. Move the specific public sources where you see failures you cannot otherwise explain to ARC Fast, which also adds residential exits with city targeting.
Sources & review
Last reviewed by the ARC Proxies team. Prices are read from the live catalog at build time.
- [1]Reuters Institute for the Study of Journalism: Digital News Report 2025: Executive summary
- [2]European Commission: FAQs: DSA data access for researchers
- [3]IETF: RFC 9309: Robots Exclusion Protocol
- [4]EUR-Lex: Regulation (EU) 2016/679 (General Data Protection Regulation), Article 4(1)
Market research
More on market research
Get started
Buy a gigabyte and try it on your own targets.
No subscription, no sales call. Code WELCOME15 takes 15% off your first purchase, and whatever you do not use is still there next quarter.