A metasearch engine shows a Lisbon to Bogota itinerary at 412 EUR. A user clicks through, lands on the carrier's own booking flow, and sees 511 EUR. Support tickets pile up, the OTA gets accused of bait pricing, and the data team gets asked an uncomfortable question: was the fare wrong, or was our collection layer wrong?
Usually it is the collection layer. Airline pricing is not a static catalogue that you scrape once and store. It is a set of live offers computed per request, shaped by the point of sale the request appears to come from, the currency, the device, the session history, and how many times that same origin has asked for the same route in the last hour. Get the request context wrong and you do not just get blocked. You get a fare that is technically real but commercially useless.
This article covers how travel pricing actually varies, what airline and OTA bot management stacks look for, and how to design an aggregation pipeline that returns fares a customer can actually book.
Why Airline Fares Are a Moving Target
Revenue management systems reprice inventory continuously. A single route may have twenty or more fare buckets, and the availability of each bucket changes with load factor, competitor moves, departure proximity, and demand signals from search traffic itself.
On top of that sits point of sale logic. Carriers publish different fares for different markets, and the same flight can carry a different base fare, tax treatment, and ancillary bundle depending on whether the request originates in Germany, Brazil, or Singapore. That is not a trick played on scrapers: it is decades-old distribution practice tied to filed fares, local competition, and currency risk. Residency-based discounts, domestic-only promo fares, and market-specific bundles all behave the same way.
Then there is the shift from legacy GDS distribution to NDC. Under NDC, the carrier returns a dynamically constructed offer with an offer identifier attached. That offer is tied to a session and often to a short validity window. A fare you collected outside a coherent session may simply not be repriceable, which is why so many aggregators report high look-to-book leakage without understanding the cause.
The practical consequence: fare accuracy depends on whether your requests look like a plausible traveller in a plausible market, not just on whether they succeeded.
What Travel Sites Actually Detect
Airline and OTA sites are among the most aggressively defended properties on the public web, because search traffic is expensive for them. Every availability query hits a pricing engine, and those engines cost real money to run. Carriers therefore have a direct financial incentive to filter non-booking traffic.
Look-to-book ratio
The metric that matters most to distribution teams is look-to-book: how many availability queries produce a ticket. Screen-scraping traffic pushes that ratio into the thousands to one. Defences are tuned to find the origins responsible. This is why a single IP that queries forty city pairs in five minutes gets throttled even if every request is technically well formed.
IP reputation and ASN classification
Commercial bot management platforms such as Akamai, DataDome, and Cloudflare's bot products classify traffic partly by network origin. Requests arriving from hosting and cloud ASNs are treated as presumptively automated on travel properties, which means clean datacenter ranges still hit interstitials and hard blocks quickly. Consumer broadband and mobile carrier ranges carry a different reputation baseline because that is where real ticket buyers sit.
Session and itinerary coherence
Modern booking flows track the whole funnel: search, results view, fare family selection, ancillaries, and payment. Jumping straight to a deep-linked results URL with no prior search, no cookies, and a fresh IP on every request is a coherence failure. Systems that watch session graphs pick that up faster than they pick up header anomalies.
Geo and currency mismatch
If the request presents a Portuguese locale, a euro currency preference, and an IP allocated in Ohio, the site has two choices: serve the US point of sale (giving you a fare your Lisbon users cannot buy) or flag the session. Both outcomes damage the dataset.
Designing an Aggregation Pipeline That Holds Up
Build a point of sale matrix, not a target list
Treat each combination of market, currency, and language as a separate collection lane. A route is not one data point: Frankfurt to New York priced from a German point of sale in euros and from a US point of sale in dollars are two distinct offers with distinct validity. Your storage schema should record the observed point of sale alongside the fare, or you will silently blend incomparable numbers into the same price history.
This is also where geo-coverage becomes a data quality requirement rather than a convenience. To publish accurate fares for fifteen markets, you need exit points that are genuinely resident in those fifteen markets.
Pace requests against human search behaviour
Human fare shoppers do a handful of searches, compare, leave, and come back later. Effective pipelines mimic that rhythm at the session level: a small cluster of related queries per session, sensible dwell time between them, and then session retirement. Volume comes from breadth of sessions, not depth per session.
Match session lifetime to the booking funnel
For availability collection, short sessions are fine. For anything that needs to survive a repricing step or an NDC offer lookup, the IP must hold for the full duration of the funnel. Rotating mid-funnel invalidates the offer and produces phantom price changes in your dataset. Static or sticky sessions solve this; aggressive rotation does not.
Prioritise routes by volatility, not by popularity
Most aggregation budgets are wasted refreshing fares that have not moved. Instrument your own data: measure how often each route and departure window actually changes price, then allocate crawl frequency accordingly. Long-haul premium cabins ninety days out move slowly. Short-haul economy inside a fourteen-day window moves constantly. Volatility-weighted scheduling routinely cuts request volume by half while improving freshness where it counts.
Validate against bookability, not against HTTP 200
A response code tells you the request went through. It does not tell you the fare is real. Sample a small percentage of collected fares and carry them one step deeper into the funnel to confirm they still price. That bookability sample rate is the single most useful quality metric a travel data team can track.
Common Mistakes That Break Fare Datasets
Treating cached fare feeds as live pricing. Cached schedule and fare data is cheap and fast, and it is fine for trend charts. It is not fine for a booking surface. Mixing the two without labelling which is which erodes user trust.
Rotating IPs on every single request. It feels safer. On travel sites it is the opposite: it destroys session coherence and makes every request look like a brand new anonymous visitor with no funnel history.
Ignoring calendar and flexible-date endpoints. Many teams brute-force one date at a time when the target exposes a month-view or flexible-date response that returns thirty data points in one call. Fewer requests, less exposure, better coverage.
Collecting the base fare only. Bags, seats, and payment fees can move the true total by 40 percent or more on low cost carriers. A fare comparison that ignores ancillaries is not a comparison.
Scaling before measuring block rates per market. Block behaviour differs sharply by carrier and by point of sale. Aggregate success rate hides the fact that one market is failing 60 percent of the time.
Where Proxies Fit In for Travel Price Intelligence
Everything above depends on request origin. You cannot observe a Brazilian point of sale from a German exit node, you cannot maintain funnel coherence without session stability, and you cannot sustain multi-market collection from a narrow range of hosting IPs that travel bot management already classifies.
That is why serious fare aggregation runs on residential and mobile pools. Requests arrive from consumer broadband and carrier networks, which is where actual ticket buyers live, and the point of sale resolves to the market you intended. For funnel work that needs continuity, sticky sessions or ISP-class addresses keep the same identity across search, select, and reprice. For high-volume cached comparisons where trust requirements are lower, datacenter capacity remains the cost-efficient option.
This is the tier of infrastructure EnigmaProxy is built for: multiple pool types under one account (residential, ISP, datacenter, and mobile), city and country level targeting across a wide geographic footprint, configurable session duration so the funnel does not break mid-reprice, and ethically sourced peer networks with documented consent. For travel teams, the practical advantage of running rotating residential proxy pools alongside static options is that one collection engine can serve both the fast breadth work and the careful funnel work without stitching together separate vendors.
Budget matters too, because fare aggregation is inherently repetitive traffic. Modelling cost per successful, bookable fare rather than cost per gigabyte is the right framing, and transparent pricing tiers make that model easier to build before you commit to volume.
Strategic Outlook for Travel Data Teams
NDC pushes collection toward sessions, not pages. As carriers move to offer-based distribution, the unit of value becomes a valid offer identifier tied to a session context. Infrastructure that can hold an identity for the life of an offer will outperform infrastructure optimised purely for request throughput.
Continuous personalisation is coming to fares. Carriers are experimenting with offers shaped by loyalty status, prior search behaviour, and device signals. Aggregators will need to report a distribution of observed prices per market rather than a single number, and that requires deliberately diverse collection identities.
Agentic booking assistants will multiply query volume. AI travel agents that search on a user's behalf will push look-to-book ratios further out of balance, which means defences will tighten again. The teams that survive will be those whose traffic is paced and geographically plausible, not those with the largest raw request budget.
Compliance scrutiny is rising. Regulators in the EU and elsewhere are increasingly interested in price transparency and drip pricing. That works in favour of well-run aggregators, provided their data collection is documented, respectful of rate limits, and running on networks with a clean sourcing story.
Conclusion
Accurate travel pricing is an infrastructure problem disguised as a scraping problem. Fares vary by point of sale, offers are session-bound, and defences are tuned to punish exactly the traffic pattern that naive aggregation produces. The fix is not brute force: it is a point of sale matrix, human-plausible pacing, session lifetimes matched to the booking funnel, volatility-weighted scheduling, and bookability sampling to prove the numbers are real.
Underneath all of that sits the network layer, and it is the part teams most often underinvest in. A provider offering genuine pool diversity, granular geo-targeting, session control, and ethical sourcing turns fare aggregation from a constant firefight into a measurable pipeline. EnigmaProxy sits in that professional tier, and for travel data teams building for the long term, that reliability is what keeps published fares matching the fares customers actually pay.