E-commerce product data scraping: attributes, GTINs and variants

GTIN, normalised to 14 digits

The primary matching key across retailers. Validate the check digit before trusting it.

Brand and manufacturer part number

The fallback key where the GTIN is missing or suspect. Normalise case and punctuation in the MPN.

Title and description

Content quality and search analysis. Store a hash so you can tell when the copy changed without diffing full text every day.

Category path

The retailer's own taxonomy, mapped to yours. Shows where a product is placed and how assortments are structured.

Attributes as key–value pairs

Size, colour, material, dimensions, energy class. Map keys to a controlled vocabulary and keep units explicit.

Variant group and variant axes

Which products belong together and what distinguishes them. schema.org calls these ProductGroup and variesBy.

Availability per variant

In stock, out of stock, pre-order or discontinued, with the date. The fast-moving field; refresh it more often than the rest.

Manufacturer details and safety information (EU)

Name and contact of the manufacturer or EU responsible person and any warnings, shown on EU listings under the General Product Safety Regulation.

1. 01

   Discover product URLs from sitemaps and categories

   Read each storefront's product sitemaps where they exist and fill gaps from category listings. Keep one URL list per country storefront.

2. 02

   Parse structured data before HTML

   Extract JSON-LD from the product page and read Product, ProductGroup, hasVariant, gtin and offers.availability. Fall back to HTML selectors only for fields the markup omits.

   Read schema.org product data through a country exit24 linescopy

   import jsonimport httpxfrom selectolax.parser import HTMLParserGATEWAY, PASSWORD = "proxy.arcproxies.io:8000", "PASSWORD"def fetch_product(url: str, country: str) -> list[dict]:    user = f"USERNAME_pool-arc_country-{country}"    with httpx.Client(proxy=f"http://{user}:{PASSWORD}@{GATEWAY}", timeout=30) as c:        html = c.get(url).text    items = []    for node in HTMLParser(html).css('script[type="application/ld+json"]'):        try:            data = json.loads(node.text())        except ValueError:            continue        for d in data if isinstance(data, list) else [data]:            if d.get("@type") in ("Product", "ProductGroup"):                items.append(d)    return itemsdef gtin14(code: str) -> str:    digits = "".join(ch for ch in code if ch.isdigit())    return digits.zfill(14)  # GS1: store every GTIN as 14 digits

3. 03

   Resolve variants explicitly

   Where variants are not listed in the markup, request each variant's own URL or parameter, holding one sticky session per product so the variant selector and stock state stay consistent.

4. 04

   Match products across retailers

   Match on validated GTIN first, then on brand plus normalised MPN, then on title and attribute similarity for review. Keep match confidence as a field rather than forcing every product into a group.

5. 05

   Refresh by field, not by page

   Recheck availability daily from category or search listings, which show many products per request. Revisit full product pages weekly, or when the listing shows a new title or image.

Prices are list prices in EUR, excl. VAT. Volume discounts apply automatically at checkout — full ladder on the pricing page.