On the difficulty of predicting the future (2001) [pdf] | Hacker News Reader

On the difficulty of predicting the future (2001) [pdf]

library.rumsfeld.com

70 pointsaleyan30 comments

Predicting the future and future forecasting are different things.

There are many forecasting techniques available. The goal is not see into the future, but to see the likely distribution of outcomes that come from decisions and sequence of future decisions.

Rumsfeld & Co had idea of what they wanted to happen, and then they twisted everything to justify it. That's just being an idiot. Rudimentary expert-opinion forecasting would have helped. Expert opinion was specially avoided or ignored.

Philip Tetlock in his masterwork Superforecasting does a great job of showing “expert” forecasting is no better than reasonably informed “amateurs.” A must read.

Philip Tetlock on Superforecasting, on EconTalk podcast, for those interested:

http://www.econtalk.org/philip-tetlock-on-superforecasting/

Superforecasting focuses point forecasting and uses the simplest forecasting methodologies to predict outcomes. Expert-opinion methods are very just one tool for forecasters. But even Tetlock has later demonstrated multiple techniques that improve expert-opinion forecasting accuracy dramatically.

Many similar and more advanced techniques have been used in the military and business for ages. Training exercises , simulations, games, cross-impact analysis, policy capture analysis, scenario methods etc.

As a former member of the military who went through very good and relevant training, I can attest to the TACTICAL value of simulations and war games. However, they were basically worthless in training to a highly dynamic strategic environment. Many were designed to validate a system or belief rather than test the unknown. In theory war games should be built to be free play oriented...in reality the aren’t.

Those are usually just exercises with adversarial component.

The war games I'm thinking of are those what flag officers and staff do in RAND and other places. For example, they bring in diplomats and other outsiders to play the opponents. Or they may be pen and paper games (spreadsheets) scenario analysis where sequence of actions are played out and consequence of every decision are updated into the environment.

As someone who has spent years trading in forecasting tournaments, I worry this characterization might be misinterpreted as saying know nothingism is correct and we should give up on predicting the future. (Not that you meant it that way, of course.)

It's true that "amateurs" can consistently beat "experts". But not all amateurs - only some. This suggests to me there is true prediction expertise, but our system that attaches labels of "expert" does not select for it.

Prediction is very much a skill and some people are very much better than others. We just don't regularly measure or select for it explicitly.

What about the complaint/criticism that Brier scores (which Tetlock uses to score forecasters) do not weight the severity/importance of the result eg: forecasting the outcome of a war and forecasting the winner of the super bowl are weighted equally.

Totally valid criticism. I think nuances like these were what Tetlock had toughest time articulating in Superforecasting and his public interviews. He chose to emphasize the big, clear points rather than subtler, more complicated issues like how questions were chosen and how they should be compared or weighted.

Unknown unknowns are definitely difficult to predict.

https://en.m.wikipedia.org/wiki/There_are_known_knowns

Yes I just can't see it happening.

Reading that quote by Rumsfeld only reminded me of Joey Fromm friends: https://youtu.be/LUN2YN0bOi8

Worst, however, may be the unknown knowns.

Psychoanalytic philosopher Slavoj Žižek says that beyond these three categories there is a fourth, the unknown known, that which we intentionally refuse to acknowledge that we know: "If Rumsfeld thinks that the main dangers in the confrontation with Iraq were the 'unknown unknowns', that is, the threats from Saddam whose nature we cannot even suspect, then the Abu Ghraib scandal shows that the main dangers lie in the "unknown knowns"—the disavowed beliefs, suppositions and obscene practices we pretend not to know about, even though they form the background of our public values."

Care to share the source? Thanks in advance.

http://www.lacan.com/zizekrumsfeld.htm

They just quoted the wikipedia article (without attribution). It, in turn, cites the link above.

I believe in that speech Žižek was just referring to assumptions -- i.e. the things we don't know we assumed, but are acting on subconsciously. The famous slide of how to invade Iraq made the invasion look simple and definable -- but in fact that assumption proved very wrong.

" just referring to assumptions -- i.e. the things we don't know we assumed, but are acting on subconsciously"

This is much, much more than "just assumption" to Zizek. In fact, this is what he refers to as ideology and it's basically the main subject he is known for

Sure, assumptions are shorthand ideology for the brain.

Invading Iraq was easy, the problems all arose at the “okay, now what?” level.

That's part of the classic formulation, which dates to the 1960s, and which I first encountered in the 1980s in print. It's effectively a "consultant's matrix' of "known" and "unknown", giving a matrix of four elements: {KK, KU, UK, UU}.

The notion of unknown knowns strongly resembles the apocryphal Mark Twain quote; "It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so". (There's no apparent proof he'd said or written this.)

https://quoteinvestigator.com/2018/11/18/know-trouble/

By 2010, the administration I wrote this memo for would totally squander America's opportunity to lead the free world. Not learning from the US-MX war of 1840s or the Vietnam war of 1960s, we embarked on a costly and disastrous military initiative in Iraq. We destroyed privacy with the Patriot Act and crashed the global economy in 2008.

When something bad happens - take a plane crash for example - its never just one thing you can point to and say that was 100% to blame. There's always a series of events that compound together - decisions and shortcuts made years or decades before a catastrophe occurs that all work up to the event.

I could just as easily blame the British for the Balfour declaration... which led to the current Mideast strife - which led to AlQaeda causing 9/11 which lead to the military interventions in Iraq.

My point is - its too facile to blame the Bush administration for everything. Look further back...

That sort of feels like trying to justify poor choices via a belief in magical predetermination to me, though.

Al Qaeda being caused by whatever they were caused by has absolutely no bearing on the actual decisions made after their attacks on 9/11.

9/11 happening doesn't determine what the outcome of 9/11 will be in terms of policy response.

The response of the Bush administration was important, and led to many (I suspect intentional) negative outcomes for the American public in general.

To just wave it away and say - it comes from earlier in history - strikes me as a bad idea at best, and a way to avoid responsibility for your actions at worst.

So who's responsible for the poor execution of the military initiative in Iraq? Was it predetermined to be a failure or could there have been a more successful outcome?

Predicting events is one thing, but there are certainly better ways to predict the results of policies.

"Predictificationism has lots of difficultants."

That prez made words too long, the next spoke them too fast; now we got one that makes them too short.

"For if there is a god it is the future. None can define it while many presume to speak on its behalf. All wish for it to deliver unto them, yet within it lies the destruction of every last one of us." - Richard Caldwell

i think a case can be made for a new, permanent status quo: America on top followed by China. Continued world peace. Continued dominance of US dollar. Contused low inflation in the US, continued tech innovation, continued dominance of Facebook, Amazon, and Google, etc.

Reply on news.ycombinator.com