News monitoring per country and language

Canonical URL

The identity of the article. Resolve tracking parameters and AMP variants to the canonical link the publisher declares.

Headline and standfirst

What readers see first, and the text most mention analysis runs on. Keep each revision with the time you saw it.

Published and updated timestamps

Lets you order coverage and measure how fast a story spread. Store the publisher's values and your own first-seen time separately.

Publisher, edition and country

The edition you collected from, not just the domain. Needed to compare coverage between markets honestly.

Language

Detected on the text, not assumed from the domain. Multilingual countries publish in several languages on one site.

Section and tags

Business, politics or local news. Puts a mention in context and filters out the sports page that happens to share a company's name.

Paywall flag

Whether the full text was available to a logged-out reader. Reach estimates should treat paywalled and open articles differently.

Short extract around the mention

Enough to classify sentiment and relevance. Keep it short and link to the source for the rest.

1. 01

   Build a source list per market

   List the publishers that matter in each country and language, and for each one find its RSS feeds and news sitemap. Note which publishers serve a different edition by location.

2. 02

   Poll feeds and sitemaps from the publisher's country

   Fetch feeds and sitemaps through a country exit every 10 to 30 minutes, using conditional requests so an unchanged feed costs almost nothing. New URLs go to the article queue.

   Conditional feed polling through a country exit19 linescopy

   import httpxGATEWAY, PASSWORD = "proxy.arcproxies.io:8000", "PASSWORD"etags: dict[str, str] = {}def client_for(country: str) -> httpx.Client:    # Rotating ISP exit in the publisher's country    user = f"USERNAME_pool-arc_country-{country}"    return httpx.Client(proxy=f"http://{user}:{PASSWORD}@{GATEWAY}", timeout=20)def poll(feed_url: str, country: str) -> str | None:    headers = {"If-None-Match": etags[feed_url]} if feed_url in etags else {}    with client_for(country) as c:        r = c.get(feed_url, headers=headers)    if r.status_code == 304:        return None  # unchanged: a few hundred bytes billed    if "etag" in r.headers:        etags[feed_url] = r.headers["etag"]    return r.text  # parse new article URLs from the feed or sitemap

3. 03

   Fetch each new article once, without the extras

   Request the article HTML with images, fonts and ad scripts blocked. Extract headline, timestamps, language and the passage around your terms. Record a paywall or consent wall rather than working around it.

4. 04

   Revisit recent articles for revisions

   Re-fetch articles at intervals during their first day and store changed headlines or corrections as new versions. After a day, most stories stop changing.

5. 05

   Cluster syndicated copy

   Group articles whose body text is near-identical, using shingled hashes, and report one story with its list of outlets. Keep original reporting and syndication as separate counts.

Prices are list prices in EUR, excl. VAT. Volume discounts apply automatically at checkout — full ladder on the pricing page.