Job posting scraping for labour-market data

Job title (raw) and normalised occupation

Keep the raw string for audit and map it to a standard such as ISCO or SOC for analysis. Most of the analytical value is in the mapping.

Advertised pay range and period

Minimum, maximum, currency and whether it is hourly, monthly or annual. Pay transparency rules are making this field far more complete than it was.

Location and remote / hybrid flag

A city plus a remote flag. Remote postings listed in every city are a common source of double counting.

Employer and industry

Separates direct employers from staffing agencies, which advertise the same role for several clients.

Skills and requirements

Extracted from the description against a taxonomy such as ESCO. This is what most workforce analytics products actually sell.

First seen / last seen

Your own observation window. Time-to-close is derived from it, and only at the resolution of your re-check cadence.

Source and canonical URL

Which board or career site, and the employer's own URL where the board links to it. Needed for deduplication and for honest coverage reporting.

Contract type and seniority

Permanent, fixed-term, part-time, internship. Mixing them flattens wage and demand trends.

1. 01

   Prefer structured sources first

   Many career sites are hosted by applicant tracking systems that publish job boards as JSON. Greenhouse, for example, documents its Job Board API GET endpoints as public with no authentication. Use those, and keep HTML parsing for the boards that have no such feed.

2. 02

   Sweep board searches per market

   Partition searches by occupation and region until each returns fewer results than the board's page cap. Run each partition on one sticky exit in the market, so the search token survives pagination.

   One sticky session per search partition17 linescopy

   import uuidimport httpxPROXY_HOST, PROXY_PORT, PASSWORD = "proxy.arcproxies.io", 8000, "PASSWORD"def sticky_proxy(country: str, minutes: int = 10) -> str:    session = uuid.uuid4().hex[:8]  # new handle per partition    user = f"USERNAME_pool-arcfast_country-{country}_session-{session}_sesstime-{minutes}"    return f"http://{user}:{PASSWORD}@{PROXY_HOST}:{PROXY_PORT}"def run_partition(search_url: str, country: str, max_pages: int = 25):    with httpx.Client(proxy=sticky_proxy(country), timeout=30) as client:        for page in range(1, max_pages + 1):            r = client.get(search_url, params={"page": page})            if r.status_code != 200:                return  # retry the whole partition later on a new handle            yield r.text

3. 03

   Fetch each new posting once, then check liveness cheaply

   Fetch the full description when a posting first appears. After that, a lightweight request on a schedule tells you whether it is still live; that is what gives you a close date.

4. 04

   Deduplicate across sources

   Match on employer, normalised title, location and a description similarity score, and merge adverts into one job record. Keep every source URL on the merged record so coverage can be audited.

5. 05

   Normalise pay and occupations, then reweight

   Convert pay to one period and currency, map titles to an occupation code, and reweight counts against an official employment benchmark before you call anything a trend.

Prices are list prices in EUR, excl. VAT. Volume discounts apply automatically at checkout — full ladder on the pricing page.