Multilingual corpus collection for LLM and NLP datasets

Document text, cleaned

Main content with navigation and boilerplate removed. Store the raw HTML or a hash of it as well so cleaning can be rerun.

Detected language and script, with confidence

Document-level language ID plus script (Latin, Cyrillic, Devanagari …). Low-confidence documents are where most contamination hides.

Exit country and fetch timestamp

Which market the page was served for and when. Needed to explain regional variants and for any later takedown request.

Canonical URL and domain

For deduplication, per-domain balancing and honouring later removal requests.

Rights signals observed

robots.txt result, page-level TDM or AI-usage metadata, and licence statements seen at fetch time. This is the evidence trail for your opt-out compliance.

Content hash and near-duplicate cluster ID

Multilingual web text is heavily duplicated across mirrors and translated templates. Deduplicate before you count tokens.

Licence, where stated

Creative Commons and public-sector licences can be stored and filtered on; many datasets publish licence breakdowns.

1. 01

   Seed per language from per-market sources

   Build seed lists from country-code domains, public broadcasters, government portals, Wikipedia-linked sites and open-licence repositories in each language. Tag each seed with the countries where its language is used.

2. 02

   Check rights signals before fetching

   Read robots.txt per RFC 9309 for your crawler's user agent, and record any machine-readable text and data mining reservation the site publishes. Skip reserved content and keep the record.

   robots.txt check, then a country-targeted fetch21 linescopy

   from urllib import robotparserfrom urllib.parse import urlsplitimport httpxPROXY_HOST, PROXY_PORT, PASSWORD = "proxy.arcproxies.io", 8000, "PASSWORD"UA = "ExampleCorpusBot/1.0 (+https://example.org/bot)"def proxy_for(country: str) -> str:    user = f"USERNAME_pool-arc_country-{country}"  # rotating ISP pool    return f"http://{user}:{PASSWORD}@{PROXY_HOST}:{PROXY_PORT}"def allowed(url: str, client: httpx.Client) -> bool:    root = "{0.scheme}://{0.netloc}".format(urlsplit(url))    rp = robotparser.RobotFileParser()    rp.parse(client.get(root + "/robots.txt").text.splitlines())    return rp.can_fetch(UA, url)with httpx.Client(proxy=proxy_for("pl"), headers={"User-Agent": UA}, timeout=30) as c:    url = "https://news.example.pl/article/123"    if allowed(url, c):        html = c.get(url).text  # store with exit country, timestamp, rights signals

3. 03

   Fetch text-only from exits in the language's markets

   Rotate through a pool per country, block images, video and fonts, and keep a per-domain request rate low. Different languages can run in parallel without sharing a rate budget.

4. 04

   Identify language, deduplicate, filter

   Run document-level language ID, cluster near-duplicates across domains, and apply quality filters tuned per language rather than one English-tuned threshold for all.

5. 05

   Balance and document

   Balance per domain and per country so one large site does not dominate a language, and publish a summary of sources and opt-out handling with the dataset.

Prices are list prices in EUR, excl. VAT. Volume discounts apply automatically at checkout — full ladder on the pricing page.