When to Hold Out For a Lower Airfare | Hacker News Reader

When to Hold Out For a Lower Airfare

fivethirtyeight.com

70 pointsrrtwo31 comments

I poked around a bit on 538 on the day it relaunched, but hadn't read anything from there since then. Is this representative of the quality of articles? It seems just embarrassing as an example of data-driven journalism. They are evaluating the quality of a "big data" prediction algorithm based on 32 samples. The experimental setup is very odd, with such a short period of price tracking, a bizarre baseline strategy, and ending the trials early (over half right at the beginning!).

Or consider this gem: "For the five routes in which there was a financial benefit to waiting, Kayak successfully reduced my fare in each instance. [...] This is a sure sign of intelligence". Yes, of course if you cherry-pick 1/6 of the trials (a whole 5 of them!), the system will look effective. What kind of value is there in "analysis" of this level?

I've been checking out fivethirtyeight every few days. You are correct. The quality has gone down since relaunching. I believe this is due to the wider focus and the need to pump out articles. They still do some decent sports articles (and I assume the election prediction stuff will still be good), but overall, I'm not impressed.

I was underwhelmed as well. In addition to the items you point out, at the end the article says:

"...it still might be worth your time and energy to use airfare prediction software. Why? Because following the algorithm isn’t going to cost you more money..."

which is simply wrong, since the article explicitly gives examples of following the algorithm costing the author more money.

I think the author was basing that statement on his previous assertion that:

> If weighted equally, the prices paid, following Kayak’s algorithm, were 2 percent higher than the initial March 29 prices. Wearing my statistical hat, I’d call that a tie between the two strategies.

The author also claims that:

> in user surveys that in addition to appealing to quantitative types, another group of regular users said Farecast gave them “peace of mind.”

So if the price difference is 2% (basically a wash either way), then waiting would be a superior strategy because it "might actually relieve some of the second-guessing that occurs when you’re left to your own devices," and that peace of mind may be worth more than 2% of the ticket cost.

...or at least, I think that's the author's point.

http://krugman.blogs.nytimes.com/2014/03/23/tarnished-silver...

I think that there's a lot of value in amateur data science from an "eyeballs" perspective. Young people want more "intelligent" coverage, but not so dense that they can't understand it. The sort of nitpicking you're engaging in suggests that you're not the target audience.

This is why I'm looking forward to more and more journals becoming open access. I think that for people who are dissatisfied with 538, understanding academic publications would not be too big of a leap.

You should check out this analysis of their analysis of their story on Bechdel-test-passing movies. It highlights some of the inadequacies in their methodologies and analysis and gives a call for more openness and higher scientific standards.

http://nbviewer.ipython.org/github/brianckeegan/Bechdel/blob...

> ...ending the trials early

Note that early termination rules are pretty common in bayesian statistics, as is taking only enough samples for a reasonable likelihood ratio.

I'd rather see samples covering different time periods as well as different routes, but I'm perfectly happy updating my belief in the efficacy of Kayak's price prediction based on these results.

I'm giving it the 6 months that it took Grantland to figure themselves out before making a call on the overall quality - The Value of a Steal series has, so far, been the only really high quality piece I've read, everything else sounds like a proposal to study something interesting (or the abstract of the paper studying something interesting).

I think one of the major areas they might struggle in is decent prose. I expected the new 538 to have lengthy and interesting prose in addition to data, but most everything has been a slog to read so far. Maybe I've just been spoiled by Grantland, but I really hope it improves.

Yea, that's true. Grantland has truly excellent writers, whereas 538 may have been looking for something different than pure writing ability. Still, guys like Kirk Goldsberry could easily write for 538 in terms of technical competence.

I regularly book from Kayak, and I like their predictor.

A sensitivity setting would be nice. For example, when it says the fare is likely to increase in the next few days, does that mean increase by $40 or increase by $140? I'm more likely to give up a probable saving of $40 to be comfortable in knowing my flight is finally booked, but I would wait if it's $140. I imagine everyone's tolerance is different, which is why the sensitivity setting would be nice.

This analysis ignores the fact that it might be beneficial to buy tickets more than 14 days before your trip (an idea which always seemed like common sense to me). The idea that fares rise monotonically for the last two weeks doesn't mean that big data is dead.

tl;dr version

You will probably not save any money following Kayak’s price prediction algorithm and instead buying tickets two weeks ahead of your scheduled departure is a reasonable strategy. You should use prediction algorithms anyway, because they wont lose you money either and help you gain "piece of mind".

What about availability. If you buy sooner that's worth something.

Well the pricing predictor is trying to predict when the prices go up and go down. Availability effects the prices a lot. As prices go up, the availability tends to go down.

I'd say one nice thing about buying early is better seat selection.

If Kayak/MSFT uses machine learning for price prediction and airlines are doing the same for predicting their revenue, doesn't it result in both sides quickly reacting to each other moves and countering opponent's actions? After all, they're all using the same ticket sales data. Maybe that's why recommendation engines don't seem to work?

Not included in the cost calculations: several days of remembering to check the cost of the air fare every day, and stressing about whether or not you'll end up paying a ton when the predictor erroneously tells you to wait. The price differentials cited in this article are well worth just buying it and having it out of the way, IMHO.

Southwest has a no cost flight change feature. So monitoring is worth a lot more for them since you can take advantage of dips after you buy.

Because following the algorithm isn’t going to cost you more money, and it might actually relieve some of the second-guessing that occurs when you’re left to your own devices.

Disagree. Airfare pricing is adversarial. It used to be "common knowledge" that buying tickets on Wednesday at 3:05am Eastern was optimal. Then, airlines responded by jacking up prices at that time. You'll also get different prices from an Incognito browser than one with cookies. I'd bet you get hosed if you use your iPhone, for obvious reasons.

If something becomes commonly known as "the right way" to get fair prices, it will surely lead to unfair prices as airlines take advantage of the herd.

Because it's adversarial, the landscape is constantly changing and there are no guarantees.

Do you have any documentation on the incognito vs. cookies claim? I'm skeptical that many airlines would do this when their "advertised" price is readily available on so many other sites.

ITA's Matrix is my go-to choice. I wonder if they sell search metrics to the airlines.

I don't have any documentation, but I've been flying A LOT for many years (last year flew 108 flights for a total over over 150,000 miles). What the parent comment above you is saying is true.

It is common knowledge among frequent flyers that you can get better prices sometimes if you're not signed in. For example, if you fly 100,000 miles a year with an airlines, they're pretty confident you're going to fly with them, so why not show you a slightly higher fare? The worst part: it works. I go to HipMunk sometimes and see a fare for $500 dollars less (on the same airline), but I still book through my preferred airlines, because the benefits are worth it for me: no hassle security, no hassle baggage (first on, first off), no hassle boarding, good chance of 1st class upgrade, etc.

The good news is that if you do find a lower fare on ITA (and pretty much only ITA), you can usually call the airlines and get that fare. Fares are bucketed into fixed "fare classes" so if there is availability in a cheap bucket, the airline MUST sell it to you.

The bad news for frequent flyers is that many benefits do not extend to these cheaper fare buckets. For example, if I want even a chance to be upgraded to business class for free on a transatlantic flight, I have to purchase one of the most expensive economy fare classes. Concrete example: I am flying to Israel in a couple months; for the cheap economy fare it was $1400, but for a chance at an upgrade it was $2,100. I took the $2,100 fare, because being in business to TLV is quite important to me.

Its unfortunate, but its how it is.

Has anyone ever seen any hard evidence for the incognito/cookies price differences other than cursory anecdotal evidence? I've only ever seen "I fly a lot and everyone who flies a lot knows this is true" reasoning for this phenomenon. Has anyone ever seen/done an actual experimental study on whether or not this is a real thing?

I'm not necessarily doubting that this is a thing, I've just never seen hard evidence.

I fly a lot, and spends weeks and weeks watching prices (and not see them change). And I used to work for online travel agencies. No, this is not true. Every person who claims it is true says "I watched the prices, and waited, then when I bought the ticket it was more expensive." Completely ignoring the fact that ticket prices generally rise as you approach the time of the flight. I've seen no evidence that the person would have paid less if they used incognito mode. I've also never seen a different (well higher price) if I signed in.

Maybe you could pay some mechanical turks to do it and take screenshots for you? I've personally seen Marriott charge me more when I'm logged in, and took and posted screenshots at the time, so I really have no doubts myself. The only hard evidence you'll find are screenshots like that, but the only statistical valid numbers would be surveys people could just lie in, so I don't think you have much choice beyond a paid survey at this point for your requirements.

Do you agree that 2 weeks out is the best time to book? From what I've seen, that's still cutting it way close, and the ideal time to book is more like 4-6 weeks out

According to Kayak itself, the best time for domestic is 3-7 weeks out, and for international destinations it varies by continent: http://www.kayak.com/news/KAYAK-Travel-Guide-2014.bd.html

Perhaps.

>If you visit the airline's site multiple times, you are not penalized (price-wise) for that.

>However, given well-known yield management techniques, you are likely to see the price rise over time.

>Variations such as the modulation on the time of day make generalizable comparisons harder.

>If you browse the site in a fresh browser, you may see a different offer but the resulting price is substantially similar. This kind of makes sense as a marketing ploy — try harder to attract the comparison shopper, but sell them the same ticket in the end.

http://skeptics.stackexchange.com/questions/9597/do-ryanair-...

The last set of tickets I bought had totally different prices for cookie vs incognito mode -- ~20% higher when saddled with a cookie.

I don't know what the rationale is, but I booked a flight last month where the fare listed in ITA Matrix wasn't bookable on the airlines website (TAP Portugal), but was available on Expedia.

Some links:

* http://travel.stackexchange.com/questions/16581/does-browsin...

* http://www.quora.com/Airfares/Does-searching-for-air-tickets...

* http://studenttravel.about.com/od/planes/a/The-Students-Guid...

Short answer: they use cookies to track your searching behavior, and raise fares if you keep looking with the intent of getting you to panic buy. Correlates of high income (e.g. using a Mac or an iPhone) also hurt you.

Reply on news.ycombinator.com