Real estate listing scraping, city by city

Asking price and price history

The core series. Store every price change with the date you first saw it; reductions are one of the earliest signs a local market is turning.

First seen / last seen dates

Your own observation window, from which days on market is derived. Keep it separate from any 'listed on' date the portal prints, which can reset on relisting.

Living area, rooms, plot area

The hedonic variables a valuation model needs, and the features you deduplicate on. Normalise units at collection, not at analysis time.

Location (postcode or district, coordinates where public)

Lets you aggregate to the geography your model uses. Many portals blur exact addresses for privacy; store what is shown, not what you infer.

Property type and tenure (sale or rent)

Flats for rent and houses for sale are different markets with different turnover. Mixing them in one supply count hides both trends.

Construction year, condition, energy rating

Explains price differences between otherwise similar homes. Store the energy rating as printed, with its scale, since scales differ by country.

Listing status (active, reserved, removed)

A listing marked reserved and later reactivated is a failed sale, not new supply. The status flag is how you tell them apart.

Agent or private seller flag

Private and agency listings price and sell differently. Store the flag; leave the person's name out unless you have a reason and a lawful basis to keep it.

1. 01

   Partition each city until no search hits the cap

   Split every city into district × property type × price band searches, and check that each returns fewer results than the portal will page through. A partition that hits the cap is a partition you have to split again.

2. 02

   Sweep result pages daily from inside the city

   Fetch the result pages through a city-targeted exit and extract listing IDs and the summary price. This is the cheap pass: it tells you what is new, what changed price and what disappeared, without opening every listing.

   City-targeted result sweep on ARC Basic19 linescopy

   import httpxPROXY_HOST, PROXY_PORT, PASSWORD = "proxy.arcproxies.io", 8000, "PASSWORD"def proxy_for(country: str, city: str) -> str:    # Rotating ISP pool, city token is the city name lowercased without spaces    user = f"USERNAME_pool-arc_country-{country}_city-{city}"    return f"http://{user}:{PASSWORD}@{PROXY_HOST}:{PROXY_PORT}"def sweep(search_url: str, country: str, city: str, pages: int):    with httpx.Client(proxy=proxy_for(country, city), timeout=30) as client:        for page in range(1, pages + 1):            r = client.get(search_url, params={"page": page})            if r.status_code == 429:                break  # back off and resume this partition later            yield page, r.textfor page, html in sweep("https://portal.example/de/berlin/flats-for-sale", "de", "berlin", 20):    ...  # extract listing id, price, area; store with today's date

3. 03

   Fetch detail pages only for new or changed listings

   Open a listing's detail page when it first appears or when its summary price moves. Block images and fonts; fetch the gallery once, at first sighting, if your model uses it.

4. 04

   Deduplicate on features, not IDs

   Match listings that share location, living area within a tolerance, room count and a similar price. Treat a match that reappears after a short gap as one spell, not two, and record the gap.

5. 05

   Aggregate to an asking-price panel

   Compute median asking price per square metre, new listings, removals and median days on market per city and week. Label the series as asking prices so nobody downstream mistakes it for a transaction index.

Prices are list prices in EUR, excl. VAT. Volume discounts apply automatically at checkout — full ladder on the pricing page.