Wrong Tomorrow - holding pundits accountable for predictions | Hacker News Reader

Wrong Tomorrow - holding pundits accountable for predictions

wrongtomorrow.com

93 pointsgabrielroth34 comments

I have this mental list of "things I would love to see and am too lazy to actually implement right now and maybe someone else will come up with the same idea and spare me the work".

And now I can cross another item off of that list.

Yep, I just wish people could bet on these predictions too. Unfortunately, that is illegal in the US ...

virtual currency

That defeats the whole point, unless the virtual currency is worth something in the real world - at the very least considerable social currency or (ideally) it should be exchangeable for real goods and services.

People bet differently when they have something real at stake - ever tried to play poker with no money behind the chips? It's a completely different (and completely boring) game. It's the reason the market instantly knew who was responsible for the Challenger crash (when it took Richard Feynman and the rest of genius scientists on the panel months to even have a plausible idea).

I'd love to see a reference on that Challenger remark -- seriously.

http://www.scribd.com/doc/8142600/the-Stock-Market-Reaction-...

Sorry for the scribd link. It's the first copy I found that wasn't behind a pay wall.

The gist is that within a few hours of the Challenger crash the stock for the company responsible for the component that failed went down far more than the stocks for other companies involved in building challenger. Interestingly, the equity value the stock lost (~200 Million USD) is almost exactly the same as the estimated total loss in profit (fines, lawsuits, updates to safety standards, lost contracts, etc).

Hmmm...so traders had a 1 in 4 shot of picking the correct company and they happened to do so? I remain unconvinced. I also remain unconvinced that this data is useful. It's not like we should let trader behavior influence safety investigations, right?

Go read "The Wisdom of Crowds" - it is chock full of examples like this.

No one is saying we should use markets for safety investigations - we are saying that we should use it predicatively.

IIRC according to Feynman's memoirs the O-rings were the leading candidate already in his first meeting with NASA engineers, and in his first days in DC General Kutyna made the link to cold-weather brittleness. It did not take them months to have a plausible idea. They spent months looking into lots of things anyway.

Maybe a prediction market inside Second Life. Isn't there some kind of exchange rate for lindens to real money (http://secondlife.com/statistics/economy-market.php)?

http://home.inklingmarkets.com/

Remember: betting is always legal in the US if you call it some sort of market. Finance, how I love thee!

More specifically, they are called prediction markets. They are legal and have been around for a while. You can even bet on the weather if you like! Enron came up with the idea of Weather Futures (aka Weather Derivatives) in their heyday, and these investment instruments are still used for hedging.

As far as I know, http://www.intrade.com/ lets you bet on elections and similar current events. Can't tell you how legitimate they are though.

You are wrong.

Intrade is based in Ireland and is illegal for US users. The Iowa Markets got a special exemption from the FTC as an academic research project.

And you can't sell derivatives unless you get regulator approval (which they won't give for betting markets) (see http://hanson.gmu.edu/ideafutures.html, for example).

Hanson (a professor of Econ at GMU, and contributor at overcomingbias) is one of the strongest proponents of legal prediction markets. He (and many others) have been lobbying for them forever, with no luck.

Thanks for the correction. I clearly did not know that. In my defense, I did add "can't tell you how legitimate [Intrade] is."

What's surprising though is that there are already investment instruments for transferring risks associated with outcomes. These are essentially predictions too, but you can only go "short" on life, so to speak.

Catastrophe Bonds, the recently infamous Credit Default Swaps are all instruments that pay off based on an outcome. In other words, they aren't really linked to the price of any commodity directly.

So, what gives? What's the difference?

Even if Hanson succeeds in getting betting markets legalized, they will still be regulated into near-uselessness. Censors will preside over them, excluding entire classes of bets. For example, this kind:

http://www.outpost-of-freedom.com/jimbellap.htm

Reminds me of overcoming bias' stuff. For example: http://www.overcomingbias.com/2009/02/who-are-macro-experts....

We really need this sort of thing to take off ...

Unfortunately, the public hates them. When DARPA created a prediction market (PAM) there was so much public outrage that the head of the agency was fired.

The public hates prediction markets that try and predict terrorist attacks. Ones that make Scoble look like an ass are something else entirely.

This is extraordinarily important work to see done. Whether or not anyone is bothering to keep track of successful predictions affects the whole quality of predictions that will be publicly made, which affects the sanity of the whole society. Mod parent up and link to this project on your blog.

Are there any categories besides right and wrong? I'd imagine that the success of some predictions is open to interpretation and/or ambiguous enough to defy easy categorization. A perfect example would be "Podcasting will be a factor in the 2006 congressional elections in the US." How much counts as "a factor?" Any amount > 0? How can we be sure one way or the other about this one, without some extensive analysis?

At the very least, I think some explanation of how the determination is made should be required.

The "WRONG" ones have some commentary:

http://wrongtomorrow.com/predictions/26

Not all of them. For example:

http://wrongtomorrow.com/predictions/47

Good eye. (Not that I can think much of any counter-example, there's still no explanation.)

Very, very cool. Kind of like the Obama promise-keeper website that I thought was cool, but then promptly forgot about. I'd like to suggest a couple of features, without which I probably will promptly forget this site as well:

  1)  E-mail or alert me when a prediction is verified correct, or expires, whichever comes first.
  2)  Mirror the URL at which the prediction is made.

Great work launching the site, and best of luck to you.

If you peruse the Bloomberg wire and keep track of stock analyst predictions, this site could grow into a "forecast analyzer" of stock "experts." Of course, someone needs to login and edit whether the prediction was correct.

Actually, I remember reading sth about a hedge fund that employed just that approach. They had every analyst or broker send in their predictions or collected them from somewhere and kept track of who did well and who did not. The better someone did (the article did not say which measure the fund used), the more business the fund would send their way (in terms of order flow), betting on their recommendations to go long or short. Presumably, their portfolio would have been optimized to account for forecast accuracy, expected gains, and some measure of risk. The difficult question here would be which measure to take as "forecast accuracy", and that also touches on the site. How wrong is wrong, and how right is right? You would need some measure of dispersion to judge whether someone was right. If there is a lot of noise in some variable (say the oil price), then basically anyone will have been right sometime when the forecast horizon is long enough. On the other hand, you couldn't really say someone was "wrong" in his forecast if he said X and, by the measure of dispersion used, the actual number lies within reasonable bounds around X. Another question I already mailed the site's creator is how the variables are actually defined and tracked. Yet another question is whether and how the forecasters can adjust their forecasts. Obviously, you should be able to update your 5y forecast if tomorrow something happens that topples your assumptions. It's totally legit to do that, and I would argue such a mechanism is the real point of the otherwise pointless forecasting game.

EDIT: There is one more question I find interesting. The measure by which forecast quality is judged should also somehow account for time dimension. Concretely, if the process you are tracking conforms 9 times out of 10 with your forecast path and then "explodes" or "collapses", are you good or bad? How far does it have to collapse or explode that you are "bad", how is the situation if the granularity is finer and you can observe 100 times instead of 10 (so the process is 98 or 99 times fine and then explodes)?

One hedge fund? This is a fairly standard strategy. Analysts publish their predictions. Services like First Call aggregate them, etc.

I didn't say "one" hedge fund pursues the strategy. I said I read about one. Also, aggregators do the exact opposite of what this hedge fund was doing: averaging. Whereas the point would be to measure their predictive accuracy individually.

Actually, the data providers just give you all the data, not average anything.

I've actually worked in that business. Have you?

I, and I think lots of others, all have had this on a todo list. I think the reason they never seem to get built or take off is that there is such a lag between their establishment and their payoff.

I assume the trick is to have some dedicated submitters for a year, and then to really push the interesting right/wrong answers into the right social news sites. That should get them over the hump. I hope this site manages to do that. Bookmarked.

Related: http://www.longbets.org/

I love this in concept, but somehow feel like it'd be better established as some sort of data blog. Perhaps you could find some aspiring young writer out there willing to work for equity?

Seems to be down at the moment... and now it's ok. Weird.

I agree completely with sethg... I had this idea floating around, and it's nice to see it. Do make it easier to see some statistics, and see wrong/right predictions, as that's a bit more interesting for the casual browser.

BTW, it's really slow again.

For some reason the #s in authors, don't match up with the actual author profile.

Of all those, I tend to think T. Boone Pickens is the one who's most likely to turn out right.

The question is, if someone turns out to be right, does he get removed from the list?

Reply on news.ycombinator.com