Guide · Travel data
How to build a flight price tracker
A flight price tracker is a scheduled job that searches a fixed set of routes and dates, stores every price as a timestamped observation, compares it with the previous one and alerts when the change crosses a threshold. Most of the engineering is in two places: deciding which route-date-market combinations to query and how often, and collecting politely enough that the source keeps answering.
Reviewed 30 Oct 2026
The short answer
- Model the work as a matrix: routes × departure dates × points of sale. It grows multiplicatively, so sampling is a design decision, not an optimisation.
- One search flow should run on one IP and one session, from inside the market you are pricing.
- Store observations append-only with market, currency, fare breakdown and timestamp. Never overwrite a price.
- Alert on meaningful change — a percentage and an absolute floor — and debounce, or users get noise.
- Prefer an official API or feed where one exists; scrape public, logged-out pages only, honour robots.txt and back off on 429.
The architecture at a glance
| Component | Job | Design note |
|---|---|---|
| Scheduler | Decides which matrix cells are due | Cadence by days-to-departure; priority for watched routes |
| Fetcher | Runs one search flow per cell | Sticky session per flow, exit in the target market, per-host concurrency cap |
| Parser | Turns a response into fare rows | Prefer the JSON the page itself loads; validate every field |
| Store | Keeps every observation | Append-only, keyed by route, date, market, fetched_at |
| Change detector | Compares with the previous good observation | Ignore failed or partial fetches; threshold plus debounce |
| Alerter | Notifies users or downstream systems | Batch per user; include old price, new price and fetched_at |
Size the route-date matrix before you write code
The matrix is routes × departure dates × points of sale × (trip lengths, for returns). Fifty routes, 90 departure dates and three markets is already 13,500 searches per full pass. Add three return lengths and it is 40,500.
So decide what a pass means. Common choices: refresh near-term departures more often than far-out ones (see how often flight prices change), refresh only a rotating fraction of far-out cells per run, and refresh on demand when a user adds a watch. Each choice is a trade between freshness and request volume — and request volume is both load on the source and bandwidth on your bill.
Estimate bytes per search flow early by measuring a handful of real flows. The bandwidth guide shows how to turn page weight into a monthly figure.
Sessions and markets
A fare search is usually several requests — search, results, sometimes a fare-detail call — that share cookies or a token. If the exit IP changes in the middle, the flow often breaks. Hold one IP per flow with a sticky session and start a fresh one for the next flow.
The market must be fixed per flow too. On a gateway that targets through the username, that means one string per market and flow, for example USERNAME_pool-arcfast_country-de_session-k3f9a1x2_sesstime-10 for a ten-minute German session. The trade-offs are covered in rotating vs sticky sessions.
Storage: observations, not prices
- Key: origin, destination, departure date, return date (or null), point of sale, cabin.
- Values: total, base fare, taxes, fees, currency, carrier, flight numbers, fare class or brand if shown.
- Provenance: fetched_at (UTC), source, HTTP status, parser version, and a hash of the raw response kept for a short period so you can re-parse after a parser bug.
- Status: ok, no availability, blocked, parse error — so a failed fetch never looks like a price drop to zero.
An append-only table makes history queries trivial and makes alerts auditable: every alert points at two rows.
Change detection and alerting
- Compare against the last successful observation for the same key, not the last attempt.
- Alert when the change exceeds both a relative threshold (for example 5%) and an absolute floor in that currency, so small fares do not alert on rounding.
- Debounce: require the new price on two consecutive observations, or re-check once before alerting. Availability flickers when a seat is held and released.
- Treat “no availability” as its own event, not as an infinite price increase.
- Include the market and currency in the alert. A user comparing a German and a UK price needs to know which is which.
Collect politely, and prefer an official route
Where an airline, GDS or NDC API is available to you on reasonable terms, use it; it is more stable than any page. Where you collect public pages, keep to logged-out content, read the site's robots.txt and terms, cap concurrency per host, and back off with jitter when the site answers 429 — see handling 429 rate limits and collecting travel data responsibly.
For the collection setup itself — which pool to use per source and what a daily matrix costs — see the airline fares and fare alerts pages.
Code
Worked example
import random, sqlite3, string, timefrom datetime import date, timedelta, datetime, timezoneimport requestsGATEWAY = "proxy.arcproxies.io:8000"PASSWORD = "PASSWORD"ROUTES = [("FRA", "LHR"), ("MUC", "CDG")]MARKETS = ["de", "gb", "fr"] # points of saleHORIZON_DAYS = 60def proxies_for(market: str) -> dict: # One sticky session per search flow, exiting in the market being priced. sid = "".join(random.choices(string.ascii_lowercase + string.digits, k=8)) user = f"USERNAME_pool-arcfast_country-{market}_session-{sid}_sesstime-10" url = f"http://{user}:{PASSWORD}@{GATEWAY}" return {"http": url, "https": url}def search(origin, dest, day, market): """Replace with the public, logged-out endpoint you are allowed to query.""" with requests.Session() as s: s.proxies = proxies_for(market) r = s.get("https://fares.example.com/api/search", params={"from": origin, "to": dest, "date": day.isoformat()}, timeout=30) if r.status_code == 429: return {"status": "rate_limited"} r.raise_for_status() best = min(r.json()["offers"], key=lambda o: o["total"]) return {"status": "ok", "total": best["total"], "currency": best["currency"]}db = sqlite3.connect("fares.db")db.execute("""CREATE TABLE IF NOT EXISTS obs ( origin TEXT, dest TEXT, day TEXT, market TEXT, status TEXT, total REAL, currency TEXT, fetched_at TEXT)""")def last_ok(key): return db.execute("""SELECT total FROM obs WHERE origin=? AND dest=? AND day=? AND market=? AND status='ok' ORDER BY fetched_at DESC LIMIT 1""", key).fetchone()def due_dates(): # Sample: every day for the first 14 days, every 7th day after that. today = date.today() return [today + timedelta(d) for d in range(1, HORIZON_DAYS) if d <= 14 or d % 7 == 0]for origin, dest in ROUTES: for day in due_dates(): for market in MARKETS: key = (origin, dest, day.isoformat(), market) prev = last_ok(key) res = search(origin, dest, day, market) db.execute("INSERT INTO obs VALUES (?,?,?,?,?,?,?,?)", (*key, res["status"], res.get("total"), res.get("currency"), datetime.now(timezone.utc).isoformat())) db.commit() if res["status"] == "ok" and prev: change = (res["total"] - prev[0]) / prev[0] if abs(change) >= 0.05 and abs(res["total"] - prev[0]) >= 10: print(f"ALERT {key}: {prev[0]} -> {res['total']} {res['currency']}") time.sleep(random.uniform(1.0, 3.0)) # pace requests per hostFAQ
Questions
Something not covered? Write to contact@arcproxies.io.
How do flight price trackers work?
They search a fixed set of routes and dates on a schedule, store each price as a timestamped observation, compare it with the previous successful observation and notify the user when the change crosses a threshold. The schedule usually checks near-term departures more often than far-out ones.
How many searches does a flight price tracker need?
Routes times departure dates times points of sale, times trip lengths for returns. Fifty routes, 90 dates and three markets is 13,500 searches per full pass, which is why trackers sample far-out dates and prioritise routes users actually watch.
Why use a sticky session for flight searches?
A fare search is usually several requests that share cookies or a token. If the exit IP changes mid-flow the session often breaks. Holding one IP per search flow with a sticky session, then starting a new session for the next flow, keeps each flow coherent.
What should a flight price tracker store?
Every observation, append-only: route, departure and return dates, point of sale, cabin, total, base fare, taxes, fees, currency, carrier, fetch time, status and parser version. Overwriting prices destroys the history that alerts and analysis depend on.
How do I avoid false price alerts?
Compare against the last successful observation rather than the last attempt, require both a percentage and an absolute change, confirm the new price on a second check before alerting, and treat no-availability as its own status rather than as a price.
Is it allowed to scrape flight prices?
It depends on the site, its terms and the jurisdiction. Prefer an official API where one is available, collect only public logged-out pages, honour robots.txt and rate limits, read the site's terms, and take legal advice for your specific case.
Sources & review
Last reviewed by the ARC Proxies team. Prices are read from the live catalog at build time.
- [1]IETF: RFC 9309: Robots Exclusion Protocol
- [2]IETF: RFC 6585: Additional HTTP Status Codes (429 Too Many Requests)
- [3]AWS Architecture Blog: Exponential Backoff And Jitter
- [4]IATA: Dynamic Offers Factsheet
Next
Where to go from here
Guides