Online price index scraping for inflation nowcasting

Price, regular price and promotion flag

Store shelf price and any struck-through regular price separately. Whether to include sale prices is a methodological choice; you cannot make it later without both.

Product identifier and description

Retailer ID plus a normalised description, brand and size. Used to match the same product over time and detect relaunches.

Pack size and unit price

Shrinkflation is a price change. Unit price lets you see it when the shelf price did not move.

Category mapped to COICOP

Mapping each product to the classification used by consumer price indices lets you weight your index like the official one.

Availability / out-of-stock

A product that disappears and returns is not a new product, and stock-outs are themselves a signal of supply pressure.

Retailer, market and exit country

Which market the price was served for. The core of any cross-country comparison, including law-of-one-price work.

Collection timestamp

Lets you check that collection time did not drift, and align daily series across markets in different time zones.

1. 01

   Define the basket and the retailers per market

   Pick retailers with a large share of online sales per market and the categories you can cover well online. Map categories to COICOP so official expenditure weights can be applied.

2. 02

   Collect category listings, not just product pages

   Category listing pages carry prices for dozens of products in one request. Use them for the daily price sweep and open product pages only for new items, attribute changes and size checks.

   Daily sweep, one market per pool, fixed start time19 linescopy

   import datetime as dtimport httpxPROXY_HOST, PROXY_PORT, PASSWORD = "proxy.arcproxies.io", 8000, "PASSWORD"MARKETS = {"de": ["https://shop.example.de/c/dairy?page={n}"],           "fr": ["https://shop.example.fr/c/produits-laitiers?page={n}"]}def proxy_for(country: str) -> str:    user = f"USERNAME_pool-arc_country-{country}"    return f"http://{user}:{PASSWORD}@{PROXY_HOST}:{PROXY_PORT}"def sweep(country: str, pages: int = 10):    stamp = dt.datetime.now(dt.timezone.utc).isoformat()    with httpx.Client(proxy=proxy_for(country), timeout=30) as client:        for template in MARKETS[country]:            for n in range(1, pages + 1):                r = client.get(template.format(n=n))                yield country, stamp, r.status_code, r.text

3. 03

   Match products over time and flag relaunches

   Match on retailer ID first, then on brand, description and size for items whose ID changed. Flag a new ID with the same description and a different size or price as a candidate relaunch for review.

4. 04

   Compute the index

   Build elementary indices per category and retailer with a chained formula, then aggregate with expenditure weights. Keep sale and regular-price versions so the choice can be revisited.

5. 05

   Monitor coverage every day

   Track products collected, matched and missing per retailer per day. A sudden drop is a collection fault until proven otherwise, and the index for that day should be flagged.

Prices are list prices in EUR, excl. VAT. Volume discounts apply automatically at checkout — full ladder on the pricing page.