Travel fares · Airline fares
Airline fare scraping by point of sale
Airline fare scraping means pricing the same flight, cabin and date from every market you sell in, and storing the fare, taxes and conditions as that market's offer. Run each search from an exit inside the point of sale and hold that exit for the whole search flow.
Pay per GB · No subscription · Prices in EUR, excl. VAT
- €2.00
- per GB on ARC Fast, excl. VAT
- −44%
- at 1 TB (€1.12 / GB)
- 185+
- countries, city targeting
- Never
- expires — no subscription
Written for
- OTA pricing and content teams
- Metasearch data engineers
- Airline revenue management analysts
- Airline competitive-intelligence teams
The short answer
- The unit of collection is route × date × cabin × point of sale, not route × date. A fare without its market attached cannot be compared with anything.
- EU law requires airlines to show the final price at all times and to itemise fare, taxes, airport charges and surcharges. Store those as separate columns, not one total.
- For flights departing an EU airport, Regulation 1008/2008 Article 23(2) prohibits discrimination by customer residence. A gap between two EU markets on the same flight is a finding to document, not a premise to build on.
- Use one sticky session per search flow and rotate between flows: residential exits for fare search, the cheap rotating ISP pool for schedules and airport data.
The problem
What goes wrong without the right IPs
The market leaks in through more than the IP
Site edition, language, currency cookie and Accept-Language all signal a market. A German exit on an English-language edition with a sterling cookie left over from the last run is a mixed point of sale, and its fare belongs to neither market. Pin all four per flow and start every flow with a clean cookie jar.
Fare and taxes arrive as one number
Results pages often show a single total. Departure taxes and airport charges differ by origin, so comparing totals across routes mixes tax differences with pricing decisions. The itemised breakdown is usually on the fare-detail step; collect it there.
The results list is not the bookable price
A results list can be served from a cache while the fare-detail step re-prices live. If you only store the list price you will publish fares that no longer exist. Record both, and treat the gap as a data-quality metric.
The exit changes halfway through a search
Search, results and fare detail are usually tied together by a token or cookie. If the IP rotates in between, the flow restarts or returns an error page. The flow, not the request, is the unit a session has to cover.
The cheapest fare class sells out between runs
When the lowest fare class on a flight sells out, the price steps up to the next one. If you store only the lowest price you cannot tell a repricing from an inventory change. Keep the booking class or fare brand wherever the page shows it.
Fare search is rate-limited harder than anything else
Search is expensive to serve, so it is throttled first. Spreading a matrix across a pool and backing off exponentially on 429 responses keeps the job running.
What to collect
What you can collect
- Route and flight numbers
- Marketing and operating carrier, so codeshares are not counted twice.
- Departure date and days to departure
- Lead time explains more variance than any other field. Store it explicitly.
- Cabin, fare brand and booking class
- Separates a repricing from a sold-out bucket.
- Base fare
- The airline's pricing decision, before government and airport charges.
- Taxes, airport charges, surcharges
- Itemised under EU rules, and they differ by origin. Needed to explain cross-route gaps.
- Final price and currency
- What a traveller in that market is actually asked to pay.
- Point of sale
- Exit country, site edition, display currency and language. All four, in four columns.
- Fare conditions
- Change and refund rules and included baggage. Two equal prices with different conditions are different products.
- Observation time and session handle
- Lets you rebuild which requests made up one flow when a price looks wrong.
How it runs
How a collection run works
- 01
Define the matrix, then sample it
List routes, departure dates, cabins and markets, then tier them. Your top routes get daily coverage close to departure; the long tail gets weekly samples. How often flight prices change covers how to set the cadence.
- 02
Pin the market for each flow
Set the exit country, then match site edition, currency and language to it. Check the exit's country once per run with a geolocation test before trusting a day's data.
- 03
One sticky session per search flow
Mint a fresh session handle per flow and hold it for about ten minutes, so search, results and fare detail all run on one IP. The next flow gets a new handle and therefore, usually, a new IP.
A fresh exit per market and per flow16 linesimport secretsimport httpxBASE_USER = "USERNAME_pool-arcfast_country-de" # pool + country from the dashboard generatorMARKETS = ["de", "gb", "fr", "it", "nl"]def flow_proxy(market: str, minutes: int = 10) -> str: """Same handle = same exit IP until the flow ends.""" user = BASE_USER.replace("_country-de", f"_country-{market}") handle = secrets.token_hex(4) return f"http://{user}_session-{handle}_sesstime-{minutes}:PASSWORD@proxy.arcproxies.io:8000"for market in MARKETS: with httpx.Client(proxy=flow_proxy(market), timeout=30) as client: # search -> results -> fare detail, all on this one exit ... - 04
Capture the fare detail, not just the list
Parse the itemised breakdown and check that fare plus taxes plus charges equals the displayed total. A row that fails the sum is a parsing bug or a hidden fee, and either one is worth knowing about before you publish.
- 05
Store observations append-only
Write every observation with its point of sale and session handle; never overwrite. Market gaps, lead-time curves and sell-out events all come from history, and you cannot reconstruct history later.
Setup
Recommended setup
Fare search is stateful and market-specific. Put residential exits on the search flow and keep reference data on the cheapest pool.
| Plan | Role | Session | Targeting | Price |
|---|---|---|---|---|
ARC Fast Rotating residential | Default for fare search Residential exits in the market you are quoting, held for search, results and fare detail. Air fares are usually priced per country, so country targeting is enough. | Sticky, about 10 minutes per search flow | Country (city rarely needed for air) | €2.00 / GB €1.12 at 1 TB (−44%) |
ARC Ultra Premium residential | For the carriers your product is judged on The premium residential pool, for the handful of carriers and markets where a missing fare is visible to your users. Leave the long tail on ARC Fast. | Sticky per search flow | Country | €3.50 / GB €2.17 at 1 TB (−38%) |
ARC Basic Rotating ISP | Schedules and reference data Timetables, airport pages and route maps do not change by market. Collect them on the cheapest pool and save the residential budget for fares. | Rotating | Country | €0.75 / GB €0.59 at 1 TB (−21%) |
Prices are list prices in EUR, excl. VAT. Volume discounts apply automatically at checkout — full ladder on the pricing page.
Connect
One connection string per market
The country, city and sticky session are tokens in the username. Change the token, change the market; keep the session handle, keep the IP.
http://USERNAME_pool-arcfast_country-de_session-k3f9a1x2_sesstime-10:PASSWORD@proxy.arcproxies.io:8000Swap _country-de for any market you quote. The full grammar is on the residential proxies page.
Illustration
What a five-market fare matrix costs
An illustration with round numbers — your pages, payloads and retry rate will differ. The arithmetic is the real price ladder.
Assumptions
- Routes
- 80
- Departure dates sampled per route
- 30
- Points of sale
- 5
- Cadence
- once a day
- Weight per search flow (assets blocked)
- 400 KB
- Bandwidth per month
- ≈ 144 GB
On ARC Fast
€195.84per month, excl. VAT
144 GB bought in one purchase at €1.36 / GB (−32% volume rung, applied automatically).
The same 144 GB on ARC Ultra: €372.96 at €2.59 / GB. Route only the jobs that need it through the pricier pool.
A headless browser with images, fonts and third-party scripts blocked. Leaving them on can multiply the weight several times over.
Bandwidth never expires, so a bigger single purchase is only ever cheaper per GB. See the ladder.
In depth
Airline fares, in depth
What point of sale means for airline fare scraping
The point of sale is the market a ticket is sold in: the country edition of the airline's site, the agency's location, and the currency and conditions that go with it. Fares are commonly filed and managed per market, which is why the same seat can carry different prices in Frankfurt and in London. Why flight prices differ by country explains the mechanics in plain terms.
Do not assume every carrier prices every market differently. IATA's 2019 study of leisure distribution noted that OTAs and metasearch engines generate many queries by making requests from several points of sale, and suggested that airlines tell them when their offer management is agnostic to the point of sale. Some carriers price per market and some do not. Your data should be able to show which is which.
In practice, the market a website assigns you comes from the exit IP plus the site edition, currency and language you request. Changing the currency selector usually converts one market's fare rather than fetching another market's fare. Only a consistent set of all four signals gives you a clean observation. The travel fare aggregation overview applies the same setup to hotels and rail.
What EU law says about how airline fares are displayed
For flights departing EU airports, Regulation (EC) No 1008/2008 sets the display rules, and the Court of Justice has interpreted them several times. These rules shape what a public page shows, so they also shape your schema.
| Rule | What it says | What it means for your data |
|---|---|---|
| Art. 23(1), final price | The final price must be indicated at all times, including taxes and unavoidable, foreseeable charges, surcharges and fees | Store the final price as its own field, separate from the base fare |
| Art. 23(1), itemisation | Fare, taxes, airport charges and other charges such as security or fuel must be specified | Four columns, not one. Validate that they add up to the final price |
| Art. 23(1), optional supplements | Optional price supplements must be communicated clearly at the start of booking and accepted on an opt-in basis | Ancillaries are a separate series. See airline ancillary prices |
| Art. 23(2), non-discrimination | Access to fares must be granted without discrimination based on the customer's nationality or place of residence | Market gaps between EU points of sale on EU departures are worth flagging, not presuming |
| C-290/16 (2017) | Taxes, airport charges and other charges must be specified separately and may not be even partially included in the air fare | A fare field that moves with airport charges is a parsing problem |
| C-28/19 (2020) | Unavoidable items must be shown from the first time the price is shown, including card fees if only one approved card is free | Results-list prices on EU sites should already include unavoidable fees |
This describes what the texts say. It is not legal advice; for a specific market or display question, ask counsel.
Results list or fare detail: which price to store
- Store both. The list price is what a shopper sees first. The fare-detail price is what they can book. The difference between the two is your staleness metric.
- Treat a re-price above the list price as normal. Results pages are often cached; selection triggers a live price. Flag gaps above a threshold rather than discarding the row.
- Re-check before you publish. If your product shows a fare to users, confirm the cheapest candidates live, as a fare-alert pipeline does, before anything is shown.
- Keep the booking class. Without it, a sell-out looks like a price rise and your lead-time curves come out wrong.
Choosing the sample: routes, dates and cadence
A full matrix is rarely affordable or necessary. Most teams sample by lead time: dense close to departure, sparse far out. A starting point you can tune against your own change rates:
| Days to departure | Typical cadence | Why |
|---|---|---|
| 0–14 | Several times a day for top routes | Inventory is thinnest and buckets close fastest |
| 15–60 | Daily | The window most leisure and business bookings fall in |
| 61–180 | Two or three times a week | Prices move, but on a slower clock |
| 180+ | Weekly | Mostly schedule and initial-pricing changes |
Measure, then adjust. If a route's price changes on fewer than one run in ten, halve its cadence. If it changes on most runs, the cadence is too low for the product you are building.
Website, GDS and NDC: three sources for one fare
The same flight can be priced through a GDS, through an NDC API, or on the airline's own website. IATA describes NDC as an XML-based data transmission standard that lets airlines build and distribute offers regardless of the distribution channel. Where you can get licensed access to a carrier's offers through an API, take it: it is cleaner data, and it is the airline's preferred channel.
Public-page collection answers a different question: what does a traveller in market X see right now? For that, the airline's own logged-out booking flow is the reference. Keep collection light. IATA's study put look-to-book ratios at 100–300 to 1 on airlines' own websites against commonly above 10,000 to 1 for OTAs and metasearch, and every search you run adds to the carrier's shopping cost. Read each site's terms and robots.txt, and follow collecting travel data responsibly.
Works with anything that takes a proxy URL
Playwright- Python requests / httpx
Crawlee- Scrapy
- Node.js
- Docker
FAQ
Questions
Something not covered? Write to contact@arcproxies.io.
How do I scrape airline fares for a specific country?
Send the whole search flow through an exit IP in that country, and set the site edition, currency and language to match it. Hold one sticky session from search to fare detail so the flow does not break, then store the result with the country, edition, currency and language as separate fields.
Does changing the currency on an airline website change the point of sale?
Usually not. A currency selector typically converts the fare of the market you are already in, rather than fetching another market's fare. The point of sale follows the site edition and where the request comes from, so to observe another market you need an exit in that market as well as the matching site edition.
Is it legal for airlines to charge different prices in different countries?
Outside specific rules, market-based pricing is common commercial practice. For flights departing an EU airport, Article 23(2) of Regulation 1008/2008 requires fares to be available without discrimination based on the customer's nationality or place of residence. This is a description of the rule, not legal advice; check with counsel for a specific case.
Should I store the total price or the fare breakdown?
Both. EU rules require airlines to show the final price and to itemise fare, taxes, airport charges and other surcharges. Storing them separately lets you tell an airline's pricing decision apart from a tax difference between origins, and lets you validate that the parts add up to the total.
How much bandwidth does airline fare scraping use?
It depends mostly on the rendering method. A headless browser with images and third-party scripts blocked typically spends a few hundred kilobytes per search flow; a full page load can use several megabytes. Multiply by routes, dates, markets and runs, and you are billed per GB. The balance never expires, so a paused run costs nothing.
Residential or ISP proxies for airline fares?
Residential for the fare search itself, because you want exits that look like consumer connections in each market. The cheaper rotating ISP pool suits schedules, airport pages and other reference data that does not vary by market. Many teams run both and route each source to the pool it needs.
Do I need city targeting for flights?
Rarely. Air fares are usually set per country, so country targeting is enough. City targeting is available on ARC Basic, Fast and Ultra for listed metros such as Frankfurt, Paris or New York, and it matters more for hotel and rail prices than for air.
Sources & review
Last reviewed by the ARC Proxies team. Prices are read from the live catalog at build time.
- [1]EUR-Lex (Official Journal of the European Union): Regulation (EC) No 1008/2008 on common rules for the operation of air services in the Community
- [2]Court of Justice of the European Union: Press release No 75/17: Judgment in Case C-290/16, Air Berlin
- [3]Court of Justice of the European Union: Press release No 49/20: Judgment in Case C-28/19, Ryanair and Others v AGCM
- [4]IATA: Addressing NDC scalability challenges in the leisure market (March 2019)
- [5]IATA: New Distribution Capability (NDC)
Travel fares
More on travel fares
Get started
Buy a gigabyte and try it on your own targets.
No subscription, no sales call. Code WELCOME15 takes 15% off your first purchase, and whatever you do not use is still there next quarter.