Hacker News Reader

"and the sample set is large enough, you would -expect- there to be some outliers who mostly got things right, by pure chance."

According to the article, "In fact, she's so good she's been put on a special team with other super forecasters whose predictions are reportedly 30 percent better than intelligence officers with access to actual classified information."

If they mean that after the team was formed, the predictions are still 30% better, than random chance ceases to be an acceptable explanation. If they mean that the team that they formed was 30% better on the predictions used to grant entrance into the team, then your objection still holds. Unfortunately, I don't think the article makes it clear which is the case.

That said, there are definitely some mechanisms that could lead an outside group to higher accuracy on these sorts of things. The two biggest are: 1. The insiders are simply that bad, for whatever reason, and random chance can easily beat them, and 2. The failures of the insiders are correlated and overwhelm their theoretically superior information, and I'd especially finger potential ideological biases there, which could affect both information filtering and the analysis. (It is one thing to believe the world should be a certain way as a result of one's ideology... it is quite another to believe the world must already be a certain way as a result of one's ideology, even when one is staring evidence in the face that it is not. One is noble and laudable... the other somewhat less so.)

Its not clear they continue to be 30% better - just that they started out that way, and got put on a team.

I'm still not convinced these folks aren't just outliers. People in all walks of life give far too much credence to lucky outliers - CEOs are hired that way, ball players, stock portfolios, sports predictors. If you make enough predictions and cherry-pick the best, you can always find a 'miracle man'.

"Its not clear they continue to be 30% better"

Yes, that was my point. The article failed to make it clear. It is worded such that it could go either way.

I agree they may be outliers. However, I can't dismiss out-of-hand the possibility that they are actually better. Outliers are common... but so are systematic/correlated weaknesses within large organizations.

Total guess here, but you would imagine that since it is a project based on statistical prediction, they would continue to measure their ability to predict.

Yes I look forward to a follow-up on this group

> but so are systematic/correlated weaknesses within large organizations

Indeed, that's such a big problem that there's a sort-of famous book written by a CIA analyst, "Psychology of Intelligence Analysis", that gives a pretty clear breakdown of how and why analysts fail to reason correctly about situations.

It also included examples of how Robert Gates (yes, the later SECDEF) managed to improve analyst reporting by actually reading the reports submitted and figuratively shredding the ones which were bad.

Before that there had been effectively no downside to submitting reports which were crap, intellectually speaking, and eventually things had become just professional-sounding educated guesses.

Large agencies don't have the benefit of small groups self-critiquing their own performance, so they have to try very hard to set and enforce standards of quality. When they don't do this it should not be surprising when quality drops.

As a guess, I would suspect that the researchers who set up the experiment are more careful about their analysis than NPR is with its reporting. NPR needs to popularize its information, thus ensuring that we, the casual reader, can get a grasp of what's going on. That necessarily dumbs down the content.

Why did they go half way, though? If you're going to dumb down things, why don't just say "scientists found that" instead of giving a distorted view of the experiment, popularizing a wrong idea of what real science actually looks like?

> "According to one report, the predictions made by the Good Judgment Project are often better even than intelligence analysts with access to classified information"

This makes me wonder how well analysts do, when they aren't given classified intelligence information. It's entirely possible that the analysts' skills are being hamstrung by information they think they know, but isn't actually true.

> and I'd especially finger potential ideological biases there, which could affect both information filtering and the analysis

An excellent book on just this subject

https://www.cia.gov/library/center-for-the-study-of-intellig...

In general these are valid remarks. But we're not talking about random clueless people deciding to play statistics, the research is obviously conducted by people who know how this kind of stuff (statistics) works. This reminds me of laypeople trying to poke holes in theories of physics (and I'm not talking about quacks and lunatics, just well intentioned people who think they have a better explanation based on rudimentary high school or even undergraduate science and math). It's such a charming, cute naivete. Do they really think their simple ideas never occurred to a single professional physicist who immediately saw it was wrong for a number of reasons?

So, do you really think the researchers in this case hadn't thought of this? I mean yeah, it's OK to doubt and inquire, but come on, this is so rudimentary...

> If the questions are all yes/no questions

The article says (or at least implies) that the subjects give probabilities of "yes" to each question.

EDIT: Oh, and to add, the article clearly says they're not just making shit up and randomly coming up with probabilities. They collect open source info on events and base their estimates on that. So it's not just blind guessing.

As to the researchers probably know what they're doing, see my original comment and the very first thing I said, "I need to know more". It's not charming cute naiveté, I simply really think that the article tells me nothing about the methodology.

See response here as to why they're still yes/no questions (TLDR; the outcomes are yes/no) - https://news.ycombinator.com/item?id=7515824

Well, I didn't mean your comment was a case of naivete, because, TBH, the analogy with my layman physics example isn't perfect, but it reminded me, it's a similar thing: people who are or should be aware that they have only superficial info on something, act as though what they know is all there is to it. The point is, we can't possibly come up with a meaningful critique of this research based on the given article, and, I think, the most reasonable course of action in this particular case is to assume the researchers know what they're doing (considering who's involved).

I think the only thing to conclude from the article is that normal people can beat CIA analysts. This is similar to the story about monkey stock picking. (near useless FT story url http://www.ft.com/intl/cms/s/0/abd15744-9793-11e2-b7ef-00144...)

Hm, I wouldn't say normal people, but subpopulations of normal people (i.e. not individuals but groups). It's an important distinction in this case. But it depends on how CIA analysts' estimates are aggregated if at all. In fact it would be extremely interesting if CIA normally got predictions by averaging multiple analysts' estimates. That would mean normal people with open source intel are beating trained analysts with confidential intel! In that case, we could say normal people beat CIA analysts. But to me this article implies analysts' estimates are not taken as averages. Any CIA analysts here to set the record straight? :)

And further, more people in the average means less variance. Do they compare sub-populations of equal size?

>was given access to a website that listed dozens of carefully worded questions on events of interest to the intelligence community, along with a place for her to enter her numerical estimate of their likelihood.

If you need to know more, try reading the article? It says that they're not yes/no questions.

I'm participating in this project, as are at least a couple other HNers (surprising to no one, huh?).

I won't go into details about it because it's not my place to do so at this point (maybe after).

But I will confirm that for every response you give, you're required to enter a percentage estimate of likelihood. For example, you'd enter 90% or 72.212% or whatever on whichever question you're responding to. So there's a potential mechanism for further ranking of participants beyond the binary. The voting mechanism itself is more complicated, but again, I'll leave discussion for when it's over.

The interesting thing about this is it raises the [theoretical, in the absence of information about weighting/ranking systems] possibility professional intelligence analysts' relative underperformance against the measure is less due to inaccurately identifying high/low probability more frequently than amateur "superforecasters", and more due to professionals making the systematic error of overconfidence in the evidence they have when weighting their estimates - e.g. if the amateurs are a lot more likely to either pick 50% on events where there genuinely isn't enough information to forecast and less likely to assign single digit probabilities to events which no available evidence suggest are likely which nevertheless happen. In other words, it sounds highly plausible that if you asked simple binary questions about expected outcomes both groups would give almost identical answers and usually be correct, but the professionals are more confident when both groups are wrong.

If this is the case then it's reasonable to assume CIA's statisticians would have done the analysis and know that's the reason these "superforecasters" are better: doubt

I guess the reverse is also possible: professional intelligence analysts are systematically tending towards being overcautious and tend to pick numbers towards the middle of the range, either out of a desire not to look silly or because they're more aware of policy implications. But subjectively I'd assign that a lower probability.

That's very interesting! How did you become involved in this project?

Whether or not they happened is a yes/no though. It's not "I believe this is 60% likely to happen" "Why yes! It was exactly 60%!", it is instead "This person said it was 60% likely to happen, and it happened. That means this person was right, for a certain degree of right"; we don't actually know what that means.

It may be that they're weighting it based on confidence level (so saying 0% chance on something that happens counts against them, but 49% chance on something that happens counts against them less), but it still counts as a yes/no in that given enough people, and a random distribution of answers, you would expect a subset of people to always be right (though the 'amount' of right changes, this person said 51% chance of it happening, this person said 100% chance of it happening; they both got it right).

Agreed. How do we know that the top 1% predictors are there through skill rather than luck? Knowing more about the questions, numbers etc would help us to come to a good idea about how many people should make these predictions by chance alone, and so on.

Or, she could be a "Teela Brown" -- someone who is naturally (or unnaturally...) lucky.

http://en.wikipedia.org/wiki/Teela_Brown

I don't see why this is being down voted as someone who has read the Ringworld novels I couldn't help but think of Teela Brown and her luck.

Because chiph is offering a fictional explanation for a real world phenomenon. There have been lots of scientific studies involving randomness in fields as diverse as economics, game theory, gambling behavior, and parapsychology; if 'luckiness' was a measurable characteristic of people then you'd expect it to reliably show up in such contexts. Invoking it as an explanation where a simpler alternative exists (the idea that some people are more skilled at analyzing current affairs) is illogical.

I cant downvote, but i expected the link to point to someone real, not a fictional character.

Ok, here you go.

http://www.cnn.com/2014/04/01/us/powerball-lottery-winner/in...

And on the opposite side of the spectrum : http://en.wikipedia.org/wiki/Roy_Sullivan

I'm assuming their smart enough to know that and that the article was just glossing over the details.