U.S. Intelligence Community Explores More Rigorous Ways to Forecast Events | Hacker News Reader

U.S. Intelligence Community Explores More Rigorous Ways to Forecast Events

online.wsj.com

39 pointspetethomas16 comments

Except that we don't know how well this 'focus group' forecasting performs relative to voluminous reporting by analysts at State, CIA, RAND, Congressional Research, think tanks, etc. Often the difference between good analysis and bad is simply how successful the analyst is at getting their projections read by the 'right' people. Along the same lines, if policy makers could stop shopping for analysis that confirms their biases, much progress could finally be made in international and public policy.

I was thinking the same thing. It sounds like they just did a really good job of rooting out the people that were bad at the work and keeping the ones that were good around. In government I would be skeptical that they actually promote or fire people based on how good they are at the job.

It seems logical that keeping the people who did well at the task and dismissing those who didn't would be the way to find people with forecasting talent in a specific domain.

It's the same principle as evolutionary selection. If I want to have a stand of only unusual purple flowers whereas the usual ones are red, then save seeds of the purple ones and plant those seeds and keep selecting. Maybe it could be called a purple flower meritocracy.

It sure seems to be the case that many employers aren't distilling the most capable employees among those employed so an effective workforce doesn't evolve.

One thing that bugged me a little was the hantavirus prediction. Seems the incidence hadn't changed much over time, I wonder how hard it was to predict accurately?

Define a metric for "did well at the task". Whatever you define will be, at best, a close proxy to what you want. The downfall of an organization lies in that gap.

Evolution fails spectacularly badly at matching our intuitions for what we want. A naive optimization process will optimize exactly for the cost function, at the expense of everything else in the universe.

This is to say that what you propose is not that easy to accomplish when you get beyond the domain of words and into the implementation.

I found this an entertaining read on why allowing an evolutionary process to optimize for you does not always lead to the results you expect: http://lesswrong.com/lw/l8/conjuring_an_evolution_to_serve_y...

I'm not sure selection is a good way to think this particular problem, or management in general.

Selection operates by replicating or destroying an immutable unit (a genetic variant). People cannot be replicated, only fired. Furthermore, people are not immutable; most importantly, they can learn from their mistakes, and they can be taught.

Someone who fails at forecasting something correctly the first time might be someone who goes back, looks deeply at why they were wrong, and improves their method. Someone who succeeds might merely have gotten lucky. These people will not be identified correctly by a selection mechanism. Time and judgment, collaboration and the inculcation of expertise seem better tools in this regard.

> Except that we don't know how well this 'focus group' forecasting performs relative to voluminous reporting by analysts at State, CIA, RAND, Congressional Research, think tanks, etc.

Actually, we do. Philip Tetlock, who is mentioned in the article, spent more than a decade collecting tens of thousands of predictions from geopolitical experts. He found that humans are terrible at making predictions about complex phenomena; typically we are stomped by simple linear models or probablistic random walks.

Some humans were slightly better at predicting outcomes, but compared to simple models, they still sucked.

It didn't matter what other variables were involved. Education, access to classified information, experience, seniority, nationality, profession ... none of them mattered. A Russian politics expert is about as likely to pick Putin's next move as a financial economist who reads the New York Times.

Tetlock's book, Expert Political Judgement, is comprehensive. I reviewed it here: http://chester.id.au/2012/07/29/review-expert-political-judg...

You don't understand how DC works. On a given issue analysis and projections are produced by numerous agencies. For example a Country desk at State may deliver an accurate estimate, but another group at state may water it down, or it may be stuffed by a competing estimate from CIA. Since its politically safer to bury your head in the sand than it is to fight for an estimate, the process is ripe for manipulation. All this concept does is pass the buck without - apparently - any comparison to all the analysis produced.

Tl;dr - This solution speaks to the failings of the intel process and not the people.

Tetlock's work was based on individuals giving unfiltered predictions.

Reminds me a bit of Asimov's psychohistory [0].

[0] https://en.wikipedia.org/wiki/Psychohistory_(fictional)

Simply keeping those who successfully predict events over time will do nothing but end with one incredibly lucky person, who will then proceed to get their future predictions wrong most of the time, or at least as much as the group you started with.

I think the competitions are structured to show the difference between distinction and calibration. Submitting outlier predictions constantly eventually gives you the former but not the latter.

The easiest prediction in the short term is "Yesterday's Weather". Most of the time, most things don't change much.

It becomes more complicated as they then act on those predictions. Thus mucking up said predictions. And by doing so requires new predictions (expensive guesses).

(I just watched DaysOfFuturePast)

Hah, I worked on this the first time around 10 years ago when it was called FutureMAP. Didn't exactly end well.

I suggest that they hire Hari Seldon.

We've still a long way from Gaia, but internet + biological computing may take us to there faster than expected.

I guess they haven't did their homework on how to detect precrime.

All they need to do is hire some Asberger-y women, put them in vats of liquid, and give them some psychedelics. Ask them to predict the future.

How hard is that? You might argue that my approach is unscientific, but I would argue that is at least as scientific as their forecasting approach.

Reply on news.ycombinator.com