Travel fares · Travel metasearch
Travel metasearch data collection: freshness, caching and cost
A travel metasearch engine answers most searches from a cache and a few live calls, and stands or falls on how often the cached price is wrong. Proxies belong in the layer that verifies the cache and fills coverage gaps, not in front of every user search.
Pay per GB · No subscription · Prices in EUR, excl. VAT
- €2.00
- per GB on ARC Fast, excl. VAT
- −44%
- at 1 TB (€1.12 / GB)
- 185+
- countries, city targeting
- Never
- expires — no subscription
Written for
- Founders building travel comparison engines
- Metasearch data and platform engineers
- Vertical comparison sites (ferries, rail, niche air)
- Travel affiliate publishers
The short answer
- Look-to-book decides the economics. IATA's 2019 study found 100–300 searches per booking on airlines' own websites, against commonly more than 10,000 for OTAs and metasearch engines.
- Use partner feeds and APIs first. Use public-page collection to verify cached prices and to fill gaps where no feed exists.
- Set cache TTLs by volatility and days to departure, and treat price accuracy (cached vs live) as a first-class KPI.
- Show prices the rules allow: in the EU, the final air price at all times, with fare, taxes and charges itemised.
The problem
What goes wrong without the right IPs
Cached price ≠ bookable price
The core metasearch failure: a user clicks a cached price and lands on a higher one. Every unmeasured cache entry is a potential mismatch, and partners notice before your dashboards do.
Live search per user query does not scale
Passing every user search through to live sources multiplies shopping volume. IATA found look-to-book ratios tend to rise by a factor of 3 to 5 when an airline lets metasearch engines poll its offer system from the initial search. Someone pays for that, often you.
Coverage gaps
Direct-only low-cost fares, small ferry operators and regional rail often have no feed. A metasearch that only shows feed content quietly shows the more expensive half of the market.
Point-of-sale mismatch
Showing a price collected in one market to a user in another produces 'price changed' at click-out. Cache keys must include the point of sale, not only route and date.
Accuracy nobody measures
Without a continuous sample of cached prices re-checked live, you cannot set TTLs rationally, compare sources or defend your numbers to partners. Measuring accuracy is the reason the verification layer exists.
What to collect
What you can collect
- Cached price and its age
- What the user sees, and how old it is.
- Live verification price
- The same key re-priced from the right market on a full flow.
- Accuracy delta
- Live minus cached, per source, route type and age band.
- Cache key
- Route, dates, cabin, party size and point of sale.
- Source
- Partner feed, API or public-page collection, for per-source accuracy.
- TTL applied
- Lets you test TTL policies against observed accuracy.
- Final price and components
- Fare, taxes, charges and unavoidable fees, as display rules require.
- Deep link and market
- Click-out to the same market edition you priced.
How it runs
How a collection run works
- 01
Inventory your sources
List the partner feeds and APIs you have, such as GDS, NDC or affiliate feeds. Then list what they do not cover. Public-page collection is for that second list and for verification, nothing else.
- 02
Design the cache around the key and the TTL
Key on route, dates, cabin, party size and point of sale. Set TTLs from days to departure and observed volatility, short close to departure and long for dates months away.
A TTL policy you can tune against measured accuracy9 linesdef ttl_minutes(days_to_departure: int, volatility: float) -> int: """volatility: share of recent checks on this key where the price moved.""" if days_to_departure <= 14: base = 30 elif days_to_departure <= 60: base = 120 else: base = 360 return max(10, int(base * (1 - min(volatility, 0.9)))) - 03
Run a verification sampler
Every day, re-price a random and a popularity-weighted sample of cache entries live, from the right market, on a sticky session per flow. Store the deltas.
- 04
Fill coverage gaps
For sources with no feed, collect public prices for your most-searched keys on a schedule, within each site's terms, and mark them as collected rather than partner-supplied.
- 05
Close the loop
Tune TTLs per route type from the accuracy data, drop or escalate sources with poor accuracy, and report accuracy to partners. It is your strongest argument in any feed negotiation.
Setup
Recommended setup
Metasearch uses proxies in two narrow places: verification and gap-filling. Size them from your sample rate, not your user traffic.
| Plan | Role | Session | Targeting | Price |
|---|---|---|---|---|
ARC Fast Rotating residential | Live verification and gap-filling Residential exits in each point of sale, held for a full search-to-fare-detail flow, so the verification price is what a user in that market would see. | Sticky, about 10 minutes per flow | Country, the cache key's market | €2.00 / GB €1.12 at 1 TB (−44%) |
ARC Basic Rotating ISP | Static content: airports, stations, ports, hotel descriptions Reference content that does not vary by visitor and changes slowly. The cheapest pool per GB. | Rotating | Country | €0.75 / GB €0.59 at 1 TB (−21%) |
ARC Ultra Premium residential | Your highest-traffic keys Premium residential for the routes that carry most click-outs, where a missed verification is most expensive. | Sticky per flow | Country | €3.50 / GB €2.17 at 1 TB (−38%) |
Prices are list prices in EUR, excl. VAT. Volume discounts apply automatically at checkout — full ladder on the pricing page.
Connect
One connection string per market
The country, city and sticky session are tokens in the username. Change the token, change the market; keep the session handle, keep the IP.
http://USERNAME_pool-arcfast_country-de_session-k3f9a1x2_sesstime-10:PASSWORD@proxy.arcproxies.io:8000Swap _country-de for any market you quote. The full grammar is on the residential proxies page.
Illustration
What a daily verification sample costs
An illustration with round numbers — your pages, payloads and retry rate will differ. The arithmetic is the real price ladder.
Assumptions
- Top routes cached
- 2,000
- Dates per route
- 60
- Points of sale
- 3
- Share re-priced live per day
- 5%
- Weight per verification flow
- 450 KB
- Bandwidth per month
- ≈ 243 GB
On ARC Fast
€330.48per month, excl. VAT
243 GB bought in one purchase at €1.36 / GB (−32% volume rung, applied automatically).
The same 243 GB on ARC Ultra: €629.37 at €2.59 / GB. Route only the jobs that need it through the pricier pool.
360,000 cache keys, of which 18,000 are verified each day. Raise the sample on the keys that drive revenue and lower it on the long tail; the bill moves in proportion.
Bandwidth never expires, so a bigger single purchase is only ever cheaper per GB. See the ladder.
In depth
Travel metasearch, in depth
The three layers of a metasearch engine
| Layer | Source | Freshness | Role of proxies |
|---|---|---|---|
| Inventory | Partner feeds and APIs (GDS, NDC, affiliate) | Push or frequent pull | None; licensed access |
| Cache | Results of past searches and feed updates | TTL per key | None directly |
| Verification | Live re-pricing of a sample of cache keys | Continuous | Market-correct exits for full flows |
| Gap-filling | Public pages where no feed exists | Scheduled | Market-correct exits, within each site's terms |
The metasearch business model is referral: the user books on an airline or OTA site. Accuracy at click-out is what partners pay for, and what the verification layer protects.
Look-to-book: why every search cannot be live
Look-to-book is the number of shopping requests per booking. In IATA's 2019 study of NDC in the leisure market, airlines' own websites ran at 100–300 to 1, while OTAs and metasearch engines were at best 1,000 to 1 and more commonly above 10,000 to 1. Ratios rose by a factor of 3 to 5 when metasearch engines polled airline offer systems from the initial search.
The same study is frank about the trade-offs. Robots were believed to generate a significant share of shopping requests, though some, such as competitive watch, add value. Third-party caches tend to generate unproductive requests just to stay fresh. A metasearch engine that verifies a sample, rather than refreshing everything blindly, is cheaper for itself and for its sources.
Cache TTLs: freshness against cost
A starting policy to tune against your own accuracy data:
| Days to departure | Starting TTL | Verification sample |
|---|---|---|
| 0–14 | 15–30 minutes | High; this is where mismatches cluster |
| 15–60 | 1–2 hours | Medium, weighted by popularity |
| 61–180 | 4–6 hours | Low, random |
| 180+ | 12–24 hours | Minimal |
Then let the data move each band. If entries aged 90 minutes in the 15–60 band are still accurate 98% of the time, lengthen the TTL. If low-cost routes in the same band are only accurate 80% of the time, give them their own, shorter band. How often flight prices change has the background.
Measuring price accuracy
- Define it. For example: the share of verified cache entries whose live final price is within 1% of the cached price, per source, route type and age band.
- Sample two ways. A random sample keeps you honest; a popularity-weighted sample tells you what users actually experience.
- Verify in the right market. A verification from the wrong point of sale measures your proxy configuration, not your cache.
- Publish it internally. Accuracy by source is what you take into feed negotiations and what tells you when a gap-filling collector has broken.
Showing prices the rules let you show
If you display air prices for flights from the EU, Regulation 1008/2008 requires the final price to be indicated at all times, with fare, taxes, airport charges and other charges specified. The Court of Justice has confirmed that these items must be shown separately and may not be folded, even partially, into the fare (C-290/16).
For a metasearch, that means caching components, not just totals, and showing a total that includes unavoidable charges. It is a description of the rules, not legal advice; check your display with counsel in each market you serve.
Vertical metasearch: hotels, ferries, rail
The same architecture works outside air, with different volatility. Hotel rate shopping has more keys (property × dates × room) and slower prices. Ferry fares have few operators and almost no feeds, so gap-filling dominates. Rail and bus fares combine open timetables with yield-managed prices. Size the verification layer per vertical, from its own accuracy curve. The travel fare aggregation page covers the shared proxy setup.
Works with anything that takes a proxy URL
- Python requests / httpx
- Scrapy
Playwright
Crawlee- Go net/http
- Docker
FAQ
Questions
Something not covered? Write to contact@arcproxies.io.
How does a travel metasearch engine get its prices?
Mostly from partner feeds and APIs (GDS, airline NDC connections, OTA and hotel affiliate feeds), served from a cache. Live calls are reserved for a fraction of searches. Public-page collection is used to verify cached prices and to fill gaps where no feed exists.
What is look-to-book ratio in travel?
It is the number of shopping requests per booking. IATA's 2019 study found 100–300 to 1 on airlines' own websites and commonly above 10,000 to 1 for OTAs and metasearch engines. High ratios raise the cost of answering searches, which is why metasearch relies on caching.
How fresh should metasearch prices be?
It depends on how fast prices move. Close to departure, minutes matter; months out, hours are usually fine. Set TTLs by days to departure and observed volatility, then tune them with a live verification sample that measures how often cached prices are still right.
Should every user search go to live sources?
No. It multiplies shopping volume, latency and cost, and IATA found it can raise look-to-book ratios several-fold. Serve searches from a cache, verify a sample live, and refresh popular keys proactively.
Where do proxies fit in a metasearch stack?
In two narrow places: verifying cached prices live from each user's market, and collecting public prices where no feed exists. Both need exits in the right country and a sticky session per search flow. Neither belongs in front of every user query.
How much bandwidth does metasearch verification use?
It scales with your sample rate, not your traffic. In our illustration, verifying 5% of 360,000 cache keys a day uses about 240 GB a month. Bandwidth is billed per GB with no subscription and never expires.
Sources & review
Last reviewed by the ARC Proxies team. Prices are read from the live catalog at build time.
- [1]IATA: Addressing NDC scalability challenges in the leisure market (March 2019)
- [2]IATA: New Distribution Capability (NDC)
- [3]EUR-Lex (Official Journal of the European Union): Regulation (EC) No 1008/2008 on common rules for the operation of air services in the Community
- [4]Court of Justice of the European Union: Press release No 75/17: Judgment in Case C-290/16, Air Berlin
Travel fares
More on travel fares
Get started
Buy a gigabyte and try it on your own targets.
No subscription, no sales call. Code WELCOME15 takes 15% off your first purchase, and whatever you do not use is still there next quarter.