Business directory and company register scraping

Legal name and registration number

The registration number is the anchor for every match and every refresh. Store the register it came from.

Legal form and status

Active, in liquidation, dissolved. Status is the field that decays fastest and matters most to sales and KYB.

Registered address

For territory assignment and deduplication. A sole trader's registered address can be a home address, which is personal data.

Industry code (NACE, SIC or local)

Sector filtering for market sizing and targeting. Registers use different code versions; store the version.

Incorporation and filing dates

Age and filing recency are proxies for activity and are widely used in scoring.

Size indicators from filings

Headcount bands, turnover or balance-sheet totals where filings are public. Note the financial year they refer to.

Website and generic contact channels

Company domain and generic addresses such as an info@ mailbox. Prefer these to named individuals.

Source URL and fetch date

Provenance for every field, so a correction or objection can be traced and applied.

1. 01

   Inventory official sources per country

   For each country, check whether the business register offers an API or bulk download, its rate limits and its licence. Use those first; scrape only what has no official route.

2. 02

   Respect official API limits

   Throttle to the documented limit per application and back off on 429. Rotating IPs to exceed an official API's per-key limit is not a workaround; it is a breach of the terms.

   Token-bucket pacing for a register API (600 req / 5 min)14 linescopy

   import timeimport httpxRATE, WINDOW = 600, 300  # requests, seconds; set to the register's documented limitdef paced(urls, client: httpx.Client):    interval = WINDOW / RATE    for url in urls:        r = client.get(url)        if r.status_code == 429:            time.sleep(int(r.headers.get("Retry-After", WINDOW)))            r = client.get(url)        yield r        time.sleep(interval)

3. 03

   Collect trade directories per country

   Partition directory searches by sector and region, run each partition on one sticky exit in the country, and stop at the directory's page cap by narrowing the partition.

   Sticky exit per directory search partition10 linescopy

   import httpxproxy = "http://USERNAME_pool-arcfast_country-de_session-k3f9a1x2_sesstime-10:PASSWORD@proxy.arcproxies.io:8000"  # one handle per partitionwith httpx.Client(proxy=proxy, timeout=30) as client:    for page in range(1, 21):        r = client.get("https://directory.example.de/search",                       params={"sector": "2562", "region": "bayern", "page": page})        if r.status_code != 200:            break

4. 04

   Resolve entities on registration number

   Join directory records to register records on registration number or VAT ID, then on normalised name plus address. Keep unmatched directory records separate rather than forcing a merge.

5. 05

   Minimise personal data and set retention

   Drop personal fields you do not need, set a retention period per field, and keep a route for objections and deletion requests that propagates to every copy.

Prices are list prices in EUR, excl. VAT. Volume discounts apply automatically at checkout — full ladder on the pricing page.