Guide · Scraping practice
How to deduplicate listings: entity resolution for property, hotel and job data
Deduplicating listings means deciding which records — from one source or many — describe the same real-world thing: one flat, one hotel, one job vacancy. The practical pipeline is to normalise the fields, catch exact duplicates with a fingerprint, group candidates into small blocks, score pairs inside each block with fuzzy comparisons and hard rules, and merge matched pairs into clusters. The statistical basis is record linkage as formalised by Fellegi and Sunter in 1969: classify every compared pair as a match, a non-match, or a case for review.
Reviewed 30 Oct 2026
The short answer
- Normalise before comparing: case-fold, strip accents, unify abbreviations and units. Python's
casefold()maps the German ß to “ss”, whichlower()does not. - Run two passes: exact fingerprints first (cheap, certain), then fuzzy matching only where the fingerprint missed.
- Block before you compare. All pairs among 100,000 listings is almost 5 billion comparisons; pairs within postcode-and-room-count blocks are a tiny fraction of that.
- Hard rules — house number, floor area within a few per cent, employer — stop fuzzy-but-wrong merges that text similarity alone would make.
- Keep every source record and assign a cluster ID. Deleting “duplicates” throws away the evidence you need when a merge was wrong.
What counts as a duplicate
Define the entity first, because the rule differs by domain. It also helps to separate the entity (the flat) from the offer (an agent's listing of it at a price): two agents listing one flat at different prices are two offers of one entity, and you usually want to keep both.
| Domain | Same entity when… | Strong keys | Common traps |
|---|---|---|---|
| Real estate | Same unit in the same building | Address with house number, unit or floor, floor area, rooms | Several flats in one building; re-listing after expiry with a new ID; agent and portal copies |
| Hotels | Same property | Name, coordinates, street address, phone | Brand changes and renamings; two buildings under one name; apartments sold under the hotel's name |
| Jobs | Same vacancy | Employer, title, location, the employer's own reference | One title posted for many locations; weekly reposts; recruiters posting for unnamed clients |
Collection and use of these data sets are covered on real estate listings, job postings and hotel content data.
Step 1: normalise every field you compare
- Text: Unicode-normalise (NFKD), drop combining accents,
casefold(), collapse punctuation and whitespace. Python's documentation notes that case-folding converts “ß” to “ss”, so “Torstraße” and “Torstrasse” become equal. - Addresses: expand or strip common abbreviations consistently (“Str.”, “St”, “Ave”), separate house number and unit, keep postcode as its own field.
- Numbers and units: convert floor area to one unit, salary to one period, prices to amounts plus currency.
- Coordinates: round to a precision that matches the source's accuracy before comparing, or compare by distance.
- Phones and URLs: strip formatting; compare a hotel's own website domain rather than a booking link.
Step 2: catch exact duplicates with fingerprints
A fingerprint is a key built from normalised fields — for a flat, normalised address, rounded area and a price band. Identical fingerprints are near-certain duplicates and cost one dictionary lookup each. Syndicated copies of the same listing on several portals usually fall out here.
For long descriptions, exact hashes miss copies with small edits. Broder and colleagues' 1997 work on clustering the web defined resemblance between documents through shared word sequences (shingles), and showed that a sketch of a few hundred bytes per document is enough to estimate it. The same technique — shingling with min-hash sketches — finds listings whose descriptions were copied and lightly edited.
Step 3: block, then match fuzzily
Comparing every listing with every other grows quadratically: 100,000 listings make 4,999,950,000 pairs. Blocking limits comparisons to records that share a cheap key — postcode and room count for flats, city and star rating for hotels, employer and city for jobs. Choose blocking keys that true duplicates almost always share; a duplicate split across two blocks is never found.
Inside a block, score each pair. Python's difflib.SequenceMatcher.ratio() returns 2·M/T, where M is the number of matching characters and T the total in both strings — 1.0 for identical text. Weight several comparisons into one score, but apply hard gates first: different house numbers, a floor area more than a few per cent apart, or a different employer means a different entity, however similar the titles read.
Step 4: decide, cluster and review
- Three outcomes, not two. Following Fellegi and Sunter, pairs above an upper threshold are matches, pairs below a lower one are non-matches, and the band in between goes to review or to a stricter rule.
- Cluster with union-find. Merging matched pairs transitively turns A=B and B=C into one cluster. Watch for chaining: one bad link can join two large clusters. Cap cluster size or require each member to match a central record.
- Tune on labelled pairs. Label a few hundred candidate pairs by hand and measure precision and recall at different thresholds. Without that, a threshold is a guess.
Keeping clusters stable over time
- Assign each new scrape to an existing cluster where it matches; create a new cluster only when nothing does.
- Give clusters stable IDs that downstream systems can rely on, and store which source records belong to each.
- Track
first_seenandlast_seenper source record to detect re-listings: same entity, new listing ID, often a new price. - Store prices against the source record and roll them up to the cluster in queries, as described in storing price time series.
For short-term rentals listed on several platforms, the same approach applies with coordinates and photos as strong signals — see vacation rental data.
Code
Worked example
import re, unicodedatafrom collections import defaultdictfrom difflib import SequenceMatcherdef norm(text: str) -> str: text = unicodedata.normalize("NFKD", text or "") text = "".join(c for c in text if not unicodedata.combining(c)).casefold() # ß -> ss text = re.sub(r"[^a-z0-9 ]+", " ", text) # Latin-script data; adapt for other scripts return re.sub(r"\s+", " ", text).strip()def fingerprint(listing: dict) -> tuple: """Exact key: same normalised address, size and price band.""" return (norm(listing["address"]), round(listing["area_m2"]), round(listing["price"], -3))def block_key(listing: dict) -> tuple: """Cheap bucket so we only compare plausible pairs.""" return (listing["postcode"], listing["rooms"])def rel_diff(x: float, y: float) -> float: return abs(x - y) / max(x, y)def numbers(text: str) -> set: return set(re.findall(r"\d+", norm(text)))def similarity(a: dict, b: dict) -> float: # Hard gates first: different house numbers or sizes are different units. if numbers(a["address"]) != numbers(b["address"]): return 0.0 if rel_diff(a["area_m2"], b["area_m2"]) > 0.03 or rel_diff(a["price"], b["price"]) > 0.10: return 0.0 title = SequenceMatcher(None, norm(a["title"]), norm(b["title"])).ratio() address = SequenceMatcher(None, norm(a["address"]), norm(b["address"])).ratio() area = 1 - rel_diff(a["area_m2"], b["area_m2"]) price = 1 - rel_diff(a["price"], b["price"]) return 0.2 * title + 0.4 * address + 0.2 * area + 0.2 * pricedef find(parent, i): while parent[i] != i: parent[i] = parent[parent[i]] i = parent[i] return idef cluster(listings: list[dict], threshold: float = 0.85) -> list[list[int]]: parent = list(range(len(listings))) def union(i, j): parent[find(parent, i)] = find(parent, j) seen = {} for i, l in enumerate(listings): # pass 1: exact fingerprints fp = fingerprint(l) if fp in seen: union(i, seen[fp]) else: seen[fp] = i blocks = defaultdict(list) # pass 2: fuzzy, inside blocks only for i, l in enumerate(listings): blocks[block_key(l)].append(i) for members in blocks.values(): for x, i in enumerate(members): for j in members[x + 1:]: if similarity(listings[i], listings[j]) >= threshold: union(i, j) groups = defaultdict(list) for i in range(len(listings)): groups[find(parent, i)].append(i) return list(groups.values())# Two portals list "Torstraße 12" and "Torstrasse 12" with 78.0 and 78.4 m² -> one cluster;# the 95 m² flat at Torstraße 40 stays separate even with an identical title.FAQ
Questions
Something not covered? Write to contact@arcproxies.io.
What is entity resolution?
Entity resolution, also called record linkage, is deciding which records refer to the same real-world entity — a person, a company, a property — when there is no shared unique identifier. It combines normalisation, comparison of fields and a decision rule for each pair.
How do I find duplicate listings across different websites?
Normalise the fields, compare exact fingerprints, then fuzzy-match candidates that share a blocking key such as postcode and room count. Use hard rules — house number, floor area, employer — to reject look-alikes, and cluster the matches.
What similarity threshold should I use?
One you have measured. Label a few hundred candidate pairs by hand, compute precision and recall at several thresholds, and choose the trade-off your use case needs. Use a review band between a lower and an upper threshold for uncertain pairs.
How do I deduplicate job postings?
Block by employer and city, compare normalised titles and the employer's own reference where present, and treat the same title in different locations as different vacancies. Track first and last seen dates to recognise weekly reposts of one vacancy.
Is fuzzy matching too slow for large datasets?
Comparing every pair is, because the number of pairs grows with the square of the dataset. Blocking reduces comparisons to records that share a key, which keeps fuzzy matching practical for hundreds of thousands of listings.
Should I delete duplicate listings?
No. Keep every source record and assign a cluster ID. Deleting loses prices and history from the other sources, and makes a wrong merge impossible to undo.
Sources & review
Last reviewed by the ARC Proxies team. Prices are read from the live catalog at build time.
- [1]Journal of the American Statistical Association (Fellegi and Sunter): A Theory for Record Linkage (1969)
- [2]Digital Equipment Corporation SRC (Broder, Glassman, Manasse, Zweig): Syntactic Clustering of the Web (Technical Note 1997-015)
- [3]Python documentation: difflib — SequenceMatcher.ratio()
- [4]Python documentation: Built-in types — str.casefold()
Next
Where to go from here
Guides