Guides

Guides to travel data and proxies

Answers to the questions that come up before and during a collection project: why a fare differs by country, which IP type a job needs, how much bandwidth it will use and what to do about a 429.

Travel data

How fares and rates are set per market, how often they move, and how to collect them without fooling yourself.

Why flight prices differ by countryThe same flight costs different amounts in different countries because fares are filed and sold per point of sale, and taxes follow the departure airport.Do airlines raise prices when you search?What regulators and researchers found about cookie and IP tracking in airline pricing, and what actually makes a fare change between two searches.How often flight prices changeFlight prices can change many times a day: filed fares update constantly and seat availability moves with every booking. What that means for collection.How to build a flight price trackerArchitecture of a flight price tracker: route-date matrix, sampling, sessions per search, storage, change detection and alerting, with a Python sketch.Hotel parity clauses in EuropeWide and narrow hotel rate parity clauses in Europe: the German ban, French and Italian laws, the 2024 EU court ruling and the Digital Markets Act.Collecting travel data responsiblyRobots.txt, rate limits, public versus logged-in pages, terms of service, GDPR and the EU database right: a practical checklist for collecting travel data.How to scrape hotel pricesScrape hotel prices: build the stay-date grid, fix occupancy and length of stay, capture rate plans, parse structured data and store observations.Comparing prices across marketsHow to compare prices across countries: the same product, a controlled point of sale, synchronised timing, reference-rate FX and equal tax treatment.Travel price transparency rulesWhat EU, UK and US rules make travel sellers show: final prices, optional extras, prior prices, drip pricing, US hotel fees and airline fee disclosure.How airline revenue management worksHow airlines decide seat availability: fare classes, nested booking limits, forecasting, overbooking, origin-destination control and continuous pricing.How hotel dynamic pricing worksHow hotels set room rates: the best available rate, length-of-stay controls, demand signals, revenue management systems and channel managers.

Proxy basics

What the different kinds of IP are, how sessions work, and how to check what a target actually sees.

Residential vs datacenter proxiesResidential, ISP, mobile and datacenter proxies compared: where the IPs come from, how sites classify them, speed, cost and when each is the right tool.Rotating vs sticky sessionsRotating proxies change exit IP per connection; sticky sessions hold one IP for a set time. How each works, when to use which, and how to set them up.How to test a proxy's locationCheck a proxy's exit IP and geolocation with curl, why IP geolocation databases disagree, and how to verify what the target site actually sees.Proxy vs VPNA proxy relays one app's traffic, a VPN tunnels a whole device's: how they differ in scope, encryption, exit IPs and automation, and when to use each.IPv4 vs IPv6 proxiesIPv4 and IPv6 proxies compared: address space, which sites accept IPv6, why rate limits often cover a whole /64, and when an IPv6 proxy fits.Choosing a proxy providerA neutral checklist for choosing a proxy provider: IP sourcing and consent, billing and expiry, targeting, protocols, sessions, invoices, VAT and trials.How to calculate proxy costsEstimate proxy cost per month: pages × bytes per page × overhead for retries and protocol, worked scenarios in GB, and how to turn gigabytes into a budget.

Scraping practice

Bandwidth, rate limits and the habits that keep a collection job cheap and polite.

How much bandwidth scraping usesBandwidth per page for HTML-only and browser-rendered scraping, using HTTP Archive page-weight data, with a worked table for 100,000 pages.How to reduce proxy bandwidthCut proxy bandwidth by fetching HTML or JSON instead of rendering, blocking images and fonts, using compression, conditional requests and local caching.Handling 429 rate limitsWhat HTTP 429 means, how to read Retry-After, and how to retry politely with exponential backoff, jitter and per-host concurrency limits. With code.Playwright vs Puppeteer vs SeleniumPlaywright, Puppeteer and Selenium compared for scraping: languages, browsers, waiting, network control, proxy authentication and when to pick each.Web scraping with PythonWeb scraping with Python: fetch pages with requests, parse with Beautiful Soup, follow pagination, then add a proxy, retries with backoff and pacing.Storing price time seriesA schema for scraped prices: append-only observations, market, currency and session dimensions, exact decimals, idempotent inserts and change queries.Deduplicating listingsFind duplicate listings across sites: normalisation, fingerprints, blocking, fuzzy matching with hard rules and clustering, for property, hotels and jobs.Respecting robots.txtImplement robots.txt correctly: RFC 9309 matching, caching and error codes, the non-standard crawl-delay, Python's robotparser and AI crawler opt-outs.