Guide · Travel data
How to scrape hotel prices: a practical tutorial
To scrape hotel prices, query one property for many stay combinations — check-in date, number of nights and number of guests — from a fixed market, and record every room and rate plan the page returns as a timestamped observation. A hotel price only means something together with those inputs: the same room can cost a different amount per night for a one-night and a three-night stay, for one adult and for two, refundable or not, and with or without taxes and fees included.
Reviewed 30 Oct 2026
The short answer
- A hotel price is a function of check-in date, length of stay, occupancy, room type, rate plan and market. Fix all six or the numbers are not comparable.
- Size the grid before you write code: 90 check-in dates × 3 lengths of stay × 2 occupancies is 540 searches per hotel per market.
- Capture the rate plan — refundable, non-refundable, breakfast included — next to the room. The cheapest number on a page is often a different product.
- Parse from the JSON the page loads or from structured data before falling back to HTML selectors, and record whether taxes and fees are included.
- Store observations append-only with a status, so a failed fetch never looks like a sold-out night or a price drop.
What determines a hotel price on a booking page
Before collecting anything, list the inputs that change the number on the page. Each one becomes a column in your data; leave one out and two observations that look comparable are not.
| Input | Why it moves the price | What to record |
|---|---|---|
| Check-in date | Demand differs per night: weekdays, weekends, holidays, events | checkin (local date) |
| Length of stay | Per-night rates can differ by stay length; minimum-stay rules close some combinations | checkout or nights |
| Occupancy | Many rates are priced per number of adults and children | adults, children |
| Room type | Different products at the same property | Room name and, where shown, room code |
| Rate plan | Cancellation terms, board and payment timing | Rate plan name, cancellation deadline, board |
| Market and currency | Site edition and request location decide the point of sale | pos, displayed currency |
| Taxes and fees | Shown included or excluded depending on the market and site | tax_included, separate fee lines |
The market column is easy to forget because it is implicit in where the request comes from. Treat it as an input you set, not an accident of your server's location — see point of sale.
Step 1: build the stay-date grid
The grid is check-in dates × lengths of stay × occupancies × markets. Ninety check-in dates, stays of one, two and seven nights, and one or two adults give 540 searches per hotel per market. Add three markets and a 50-hotel competitive set and one full pass is 81,000 searches.
Most projects do not need the full grid every day. A common pattern is to refresh the next 14 days daily, the following weeks every few days and far-out dates weekly, and to add known event dates explicitly. Measure how often your hotels actually change before settling the cadence; each refresh is load on the site and bandwidth on your bill.
Expect gaps. A minimum length-of-stay rule can make a one-night search on a busy weekend return nothing while a three-night stay is bookable. An eCornell article on stay controls describes minimum stays as a tool for a string of busy nights followed by quieter ones. Record that outcome as no_rates for that combination — it is information, not a failed fetch.
Step 2: fix occupancy, market and session
- Set occupancy explicitly. Booking sites pre-fill a default number of guests. Put adults and children in the search parameters so a changed default cannot silently change your data.
- Choose the market deliberately. Use the site's country edition and send the request from inside that country. Residential proxies with country targeting are the usual way to collect several markets from one machine.
- One session per search flow. A search, its result list and the room detail page usually share cookies. Hold one exit IP for the whole flow with a sticky session, then start a new one for the next cell.
- Stay logged out. Member and loyalty rates sit behind a login and are out of scope. Collect what any visitor sees.
Step 3: capture rooms and rate plans, not only the lowest price
A result list typically shows one “from” price per hotel. The property page shows the real offer set: each room type with several rate plans. Collect that set, because rate shopping only works like for like — a flexible room-only rate against a flexible room-only rate.
- Room name and code, rate plan name, cancellation deadline, board (room only, breakfast), pay now or pay at the property.
- Whether the price is per night or for the whole stay, as displayed. Normalise later, never at collection time.
- Taxes and fees: included, excluded, or listed separately. In the US, the FTC's fees rule has required short-term lodging offers to show a total price including mandatory fees such as resort fees since 12 May 2025; government taxes may still be shown separately. A US page and an EU page can therefore differ in what the headline includes.
Fee handling across markets is covered in more depth on the hotel resort fees page.
Step 4: parse from structured data first
- The JSON the page loads. Open the browser's network panel while searching. Many hotel pages fetch rates from a JSON endpoint that is smaller and more stable than the HTML around it.
- schema.org markup. Some pages embed JSON-LD: a
Hotelcan carrymakesOffer, and anOfferhaspriceandpriceCurrency. Many properties publish only their description there, not live rates, so treat it as a bonus. - HTML selectors last. Use them when nothing else exists, and validate every field: a price must parse as a number, a currency must be three letters, check-in must be before check-out.
- Keep the price exact. Store it as text or a decimal, not a float, and keep the displayed currency next to it.
If the rates appear only after scripts run, you need a browser such as Playwright; it costs far more bandwidth per search than an HTTP client like Python requests.
Step 5: store observations and schedule refreshes
- Key: hotel, check-in, check-out, adults, children, market, room, rate plan.
- Values: price, currency, tax flag, fee lines, cancellation deadline.
- Provenance: fetched_at in UTC, status (
ok,no_rates,blocked,parse_error), session handle. - Never overwrite. History is what makes a rate-shopping report or a parity check possible.
Schema design, deduplication and change queries are worked through in storing price time series. How to run this as a competitive-set service is on the hotel rate shopping page.
Code
Worked example
import json, re, sqlite3, secretsfrom datetime import date, timedelta, datetime, timezonefrom itertools import productimport requestsGATEWAY = "proxy.arcproxies.io:8000"PASSWORD = "PASSWORD"# 1. The grid: check-in dates x lengths of stay x occupancies.def stay_grid(start: date, days: int, nights=(1, 2, 7), guests=(1, 2)): for offset, los, adults in product(range(days), nights, guests): checkin = start + timedelta(days=offset) yield checkin, checkin + timedelta(days=los), adults# 2. One sticky session per search flow, exiting in the market you price.def proxies_for(country: str) -> dict: user = f"USERNAME_pool-arcfast_country-{country}_session-{secrets.token_hex(4)}_sesstime-10" url = f"http://{user}:{PASSWORD}@{GATEWAY}" return {"http": url, "https": url}# 3. Parse: prefer structured data (JSON-LD) over CSS selectors.LDJSON = re.compile(r'<script[^>]+type="application/ld\+json"[^>]*>(.*?)</script>', re.S | re.I)def offers_from_html(html: str): for block in LDJSON.findall(html): try: data = json.loads(block) except json.JSONDecodeError: continue for node in data if isinstance(data, list) else [data]: offers = node.get("makesOffer") or node.get("offers") or [] for o in offers if isinstance(offers, list) else [offers]: if "price" in o and "priceCurrency" in o: yield { "room": (o.get("itemOffered") or {}).get("name"), "rate_plan": o.get("name"), "price": str(o["price"]), # keep as text; convert with Decimal later "currency": o["priceCurrency"], }# 4. Store every observation, never overwrite.db = sqlite3.connect("hotel_rates.db")db.execute("""CREATE TABLE IF NOT EXISTS rate_obs ( hotel_id TEXT, checkin TEXT, checkout TEXT, adults INT, market TEXT, room TEXT, rate_plan TEXT, price TEXT, currency TEXT, status TEXT, fetched_at TEXT)""")def collect(hotel_id: str, url_for, market: str, start: date): for checkin, checkout, adults in stay_grid(start, days=30): fetched_at = datetime.now(timezone.utc).isoformat() try: r = requests.get(url_for(checkin, checkout, adults), proxies=proxies_for(market), timeout=(5, 30)) rows = list(offers_from_html(r.text)) if r.ok else [] status = "ok" if rows else ("no_rates" if r.ok else f"http_{r.status_code}") except requests.RequestException as exc: rows, status = [], type(exc).__name__ for o in rows or [{"room": None, "rate_plan": None, "price": None, "currency": None}]: db.execute("INSERT INTO rate_obs VALUES (?,?,?,?,?,?,?,?,?,?,?)", (hotel_id, checkin.isoformat(), checkout.isoformat(), adults, market, o["room"], o["rate_plan"], o["price"], o["currency"], status, fetched_at)) db.commit()FAQ
Questions
Something not covered? Write to contact@arcproxies.io.
Is it legal to scrape hotel prices?
Collecting publicly displayed, logged-out prices is common practice, but the answer depends on the site's terms, database rights and your jurisdiction. Never work around a login, paywall or CAPTCHA, keep request rates low, and take legal advice for your specific use. This is general information, not legal advice.
Why does the same hotel cost more per night for one night than for three?
Hotels price each stay date separately and often apply length-of-stay rules. A busy night priced on its own can be expensive or closed to one-night stays, while a longer stay averages that night with quieter ones or qualifies for a different rate.
Should I store hotel prices per night or per stay?
Store exactly what the page displays together with the basis — per night or total — plus check-in, check-out and occupancy. Derive per-night or per-stay figures later. Converting at collection time loses information you cannot recover.
How many searches does it take to monitor one hotel?
It is check-in dates × lengths of stay × occupancies × markets. Ninety dates, three stay lengths and two occupancies are 540 searches per market for one full pass; most monitors refresh near dates daily and far dates less often.
Do hotel booking sites include taxes in the displayed price?
It depends on the market and the site. EU consumer prices generally include VAT. In the US, the FTC fees rule requires short-term lodging offers to include mandatory fees in the total price, but government taxes may be shown separately. Record a tax-included flag on every observation.
Can I collect member or loyalty rates?
No. Member and loyalty rates are shown only after login, and collecting behind a login is out of scope. Record the public rate a logged-out visitor sees, and note on the page if a member rate is advertised without a price.
Sources & review
Last reviewed by the ARC Proxies team. Prices are read from the live catalog at build time.
- [1]Federal Trade Commission: Rule on Unfair or Deceptive Fees: Frequently Asked Questions
- [2]eCornell (Cornell University): The Dos and Don'ts of Length of Stay
- [3]schema.org: Hotel (makesOffer, checkinTime, checkoutTime)
- [4]schema.org: Offer (price, priceCurrency, itemOffered)
Next
Where to go from here
Guides