The Experts Have Spoken | Hacker News Reader

The Experts Have Spoken

theawl.com

65 pointscolinprince12 comments

It seems silly to expect that the experts never get anything wrong. It even seems silly to expect that they don't get any of the big questions wrong. The real question is whether the ratio (or some other function) of wrong predictions to right predictions, suitably weighted by their strength and complexity, is decreasing.

My mom is sick, so I've been hanging around her house to sit with her. She has some 5-10 year old magazines, including some of one of my favorites The New Yorker. I like the intelligence of the commentators and I feel that they often put into words things that I vaguely feel.

I was stuck reading them (or else: Fox News and the Catholic channel). Boy, they get just about every prediction wrong. It was shocking.

Makes me doubt we do much better than people in the past.

We can do better, but it isn't expertise that is the difference.

It's worth taking a look at the Good Judgement Project and their success at making forecasts.

http://www.nesta.org.uk/sites/default/files/1502_working_pap... is a good overview.

So, I work in this area and follow a lot of the work on ways to make predictions (Superforecasters, etc)

One thing I see time and time again is that what we think of as predictions are really aggregates of large number of assumptions and projections of how these assumptions interact with each other and the world.

"Experts" generally are very good at estimating the likelihood of some of these assumptions changing. BUT it is only a limited number of these things they are experts in, and that make them no better in other areas.

What's your opinion of the superforecasters work?

I commented on this previously: https://news.ycombinator.com/item?id=10277039

I have read the book but I am not super convinced by the way they estimate whether a forecaster is good or not. The methodology is debatable.

> "Experts" generally are very good at estimating the likelihood of some of these assumptions changing.

Is that really true? I'm dubious, and I'm curious whether there's ever been any empirical studies.

Also, I'm not entirely clear what you mean by "some". Do you mean "some kinds of assumptions" or "some number of assumptions?

Is that really true?

I think so. For example, in fields relying on scientific progress, a scientist working in the field is generally fairly accurate in predicting the likely progress in their exact field over the next couple of years. For example in materials science the researchers are relatively good at estimating what progress will be made.

There was an IARPA program around estimating scientific progress that may have done some empirical studies.

Also, I'm not entirely clear what you mean by "some". Do you mean "some kinds of assumptions" or "some number of assumptions?

I mean they are good at estimating the likelihood of specific estimates changing.

If I had to hazard a guess, based on the venue, you probably have very concrete notions about what words mean. You hold experts to those definitions, even if they don't share them, which makes their statements seem wrong. In fact they're using a more, for lack of a better word, "parochial" dialect and in that language their predictions are useful predictors for their clients. They don't really think they're making the same claims you think they are.

Or not, I don't know. I'm not you.

There's a difference between binary and open-ended predictions. "Will the iPhone be a commercial success?" is easier than "How many iPhones will be sold?". It's the later that matters when you are directly involved. This can lead to seemingly paradox statements like "The iPhone will most likely fail, but assuming that it doesn't, it will be wildly successful".

Philip Tetlock and some controversial author have written a paper about it: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2284964

Why do all these "experts" always try to predict the future? I would much prefer if they actually gave some background about the present and the past so we can make better decisions.

Reply on news.ycombinator.com