How to collect travel data responsibly

Start with what you are collecting

Decide the minimum fields you need before you build. A fare tracker needs prices and flight identifiers; it does not need anyone's name.

robots.txt: what it means and how to honour it

The Robots Exclusion Protocol is standardised as RFC 9309. It is explicit that its rules “are not a form of access authorization” — they are the site operator's instructions to crawlers, and a responsible collector follows them.

• Fetch /robots.txt per host before collecting and match your user agent's group.
• Do not use a cached copy for more than 24 hours, unless the file is unreachable.
• If robots.txt returns a 4xx, the protocol allows access; if it returns a 5xx or the server is unreachable, assume complete disallow.
• Parse at least 500 KiB of the file; the RFC sets that as the minimum parsing limit.

Rate limits and load

Fare search is expensive to serve. Keep your request rate well below what a human-facing site is sized for, spread work across the day, and cap concurrent requests per host. When a server answers 429 Too Many Requests, it is telling you it is counting your requests; RFC 6585 notes it may count per resource, per server or across servers, and may identify you by credentials or a cookie.

Spreading requests across many IP addresses changes how a per-IP limit is accounted, not how much load the site carries. Use it to avoid concentrating a legitimate workload on one address — not to exceed what the site is willing to serve. The mechanics of a polite retry loop are in handling 429 rate limits.

Terms of service and the Ryanair ruling

On 15 January 2015 the Court of Justice of the EU ruled in Ryanair v PR Aviation (C-30/14) that the Database Directive's user rights do not apply to a database protected neither by copyright nor by the sui generis right. The practical consequence is that a website owner can restrict use of such data through contractual terms.

Whether a site's terms bind a visitor who never clicked “I agree” is a question of national contract law. Read the terms of every source, record the version you relied on, and get advice where a source's terms prohibit automated collection.

The EU database right

Article 7 of the Database Directive (96/9/EC) gives the maker of a database a right to prevent extraction or re-use of the whole or a substantial part of its contents, where there has been a substantial investment in obtaining, verifying or presenting them. Large fare, hotel and listing catalogues are typical candidates.

Collecting the specific prices you need for analysis is a different activity from rebuilding a competitor's catalogue. Keep collection proportionate to the purpose and avoid republishing a source's dataset wholesale.

Personal data and the GDPR

The GDPR covers any information relating to an identified or identifiable person, wherever it was found. Public availability does not take data out of scope. The Dutch data protection authority's May 2024 guidance went further than most: it described scraping personal data as almost always a GDPR violation and took the view that purely commercial interests do not count as a legitimate interest — a position reported to be under review.

For travel data the fix is usually simple: do not collect the personal fields. Strip reviewer names and photos, aggregate review scores, and keep host identities out of datasets about rental prices — see the vacation rental data and hospitality reviews pages for what that looks like in practice.

What we permit on our network is set out in the acceptable use policy.