Intelligent intelligence – Just how good are government analysts? | Hacker News Reader

Intelligent intelligence – Just how good are government analysts?

economist.com

73 pointskawera28 comments

For what it's worth, there is no method that goes untested for forecasting. The CIA's Sherman Kent school is a treasure trove of analytical methods, techniques and processes that have been studied and implemented over the decades. A lot of the research has been put online for free [1].

As the article states, nobody outside of the business will ever hear about 99.9% of successful intelligence work done, and 90% of the people inside won't hear about it either. So unless you are directly involved in analysis, colletions or operations, it's impossible to get a good feel for efficacy.

[1]https://www.cia.gov/library/kent-center-occasional-papers

Let me try out my media spin skills on that. ;)

"Due to a shocking lack of transparency and accountability, we can't even tell if CIA analysts are doing their job. It would appear that up to 90% of analysts collect a paycheck without producing any measurable output."

Ha, while tongue in cheek, that is something that is heard a lot within agencies.

So much so that over the years significant layoffs happen and programs get cut when higher ups in agencies don't have a proper understanding of some of their organization's capabilities.

For anyone who is interested in this topic, Dr. Tetlock just released a new book about it (http://www.amazon.com/Superforecasting-The-Art-Science-Predi...).

We're also hosting a public forecasting tournament for him and his team that focuses on geo-political forecasting: https://www.gjopen.com/

In addition - anyone interested in Tetlock's work and how to think like a super forecaster can join us at the conference this Saturday in London. https://www.eventbrite.co.uk/e/superforecasting-and-geopolit...

I'm about half-way through the book. Very easy, enjoyable read. Left wondering why some of these Amazon reviews try to make it sound as if it's quite academically/technically rigorous.

The more famous an analyst is, the more likely he is to be confident in his abilities, and the less likely to second-guess himself, to analyze his analysis. Those analysts who must defend their work to superiors do a better job.

The same thing seems to happen in other lines of work - when an engineer is accountable to code review, he might do better. When an author is subject to the will of an editor, his work is ultimately achieved faster, and with better quality; witness George RR Martin's speed on the first three books in A Song of Ice and Fire versus the glacial pace of the last two.

Your last sentence has me doubting: I think that 'faster and better' is potentially true in measurable fields with reasonably objective criteria for success (writing code is one), but it strikes me as untrue in the arts, especially regarding experimental work, which is often all about defying the norms and expectations a reviewer is in place to enforce.

If your argument was correct, we'd see authors make their best work later in their career, when they can ignore editors and publishers, because their name alone will carry their work.

I haven't done a big study in this, but it seems to me that early works are, on average, better than works made after artists have become famous. If anything, they become complacent. It's not just GRR Martin: Look at the Harry Potter series, for instance. In general, the commmercial success makes things worse.

An alternative explanation is that famous artists struggle to replicate their initial success because it often required a large amount of luck.

https://en.wikipedia.org/wiki/Regression_toward_the_mean

Authors often do get noticeably worse after becoming famous, though. Take the reasonably well-known example of Robert Jordan, who wrote a series of 5 great books followed by 5 lackluster books. It's hard to say he just got lucky five times in a row.

And you can look at the same question another way by reading self-published (therefore, unedited) books -- I used to be quite open to reading those; bitter experience has taught me not to bother.

> they were underselling themselves, tending to err more than necessary on the side of uncertainty

That's not how it works. If you consistently misreport your calibration, you're still miscalibrated. Consumers of the intelligence would (should) notice the miscalibration and correct for it, regardless of direction.

Very Interesting, though I still wonder if intelligence agencies can out perform betting markets (at least on questions with enough interest to be able to generate a liquid price).

Apparently, given that governments are cracking down on prediction markets. The conspiracy-theorist side of me could interpret it as intelligence agencies beating them quite directly.

There was an article in NPR a while back about the Good Judgment Project (which was run by Dr. Tetlock - mentioned in the article). In some cases, betting/prediction markets (& other similar tools) were beating intelligence analysts - http://www.npr.org/sections/parallels/2014/04/02/297839429/-...

I participated in GJP for a couple seasons. Honestly, I didn't do that well.

But I was approaching it very differently than I would if it was my job. Since the system that was set up gave high rewards to unpopular predictions, I just gambled on the few most unpopular that had at least some shot at reversing. It wasn't the smartest approach, but it was the most fun. If I was doing it for real, obviously I'd go a different way.

The people who did best, at least from what I saw, tended to ride waves of popularity on the more active questions, buying low and selling high.

What I did was like betting on a few biotech startups, what the best scorers did was like riding waves of the market leading stocks.

In the end, I'm not that sure it had much to do with actual prediction of events. Then again, neither did my approach. I guess I'm not sold on the version of prediction markets they were using.

Kinda depends on which platform you were on (GJP used several). Some were prediction markets while some were opinion pools, both of which are scored/rewarded differently.

Our company (Cultivate Labs) recently acquired Inkling Markets (a very early YC company that built prediction market software) and have been building a new version of the PM platform, which will hopefully address some of the risk/reward quirks.

If you're interested in this stuff, you might be interested in the two topical PM sites we're launching: -https://sportscast.cultivateforecasts.com/ (obviously focused on sports) -https://alphacast.cultivateforecasts.com/ (officially launching later this week, focused on global finance, politics, & tech).

Ah! I didn't know Inkling was a YC company. Interesting.

I'll definitely check out your new sites.

The big difference they have is that they are playing a more strategic role than the tactical short term play in most market activities - in that their predictions will drive government action which will be a major factor in any outcomes. Makes more sense to test against successful opportunism than prediction

Could there be an element of self-fulfilling prophecy explaining why intelligence analysts, on whose advice governments decide their actions, have a better success rate at prediction than others who do not so closely influence government decision making?

That assumes that intelligence agencies are not utilizing betting markets.

In fact there was a big push in 2010 to build a betting market type system for analysis across the Intelligence Community. It is not used 100% of the time but has some measure of success where applied.

When I was in the IC I actually built a hypothesis estimation tool based on the Delphi method and a Bayesian updater to give probabilities for future events, crowdsourcing the votes from the community. Worked ok in our limited run, but needed a lot more work to implement really well.

Adam Curtis' article on the competency of these people is entertaining and contains some really quite disturbing information about the incompetency of some professionals in this field.

http://www.bbc.co.uk/blogs/adamcurtis/entries/3662a707-0af9-...

I don't always agree with Curtis, but his articles and documentaries are always worth the time.

Intelligence "works" in the same sense that torture does: to further the unstated private interest of whoever hired the intelligence and torture agencies. If we're interested in the truth rather than The Truth, we can't trust non-public sources that have repeatedly failed in public ways.

I would encourage you to read the article. While I am inclined to agree with your sentiment, the article presents a study that actually deals with the concrete effectiveness of intelligence analysts.

Yeah I read the article; it wasn't very long. Either the authors of the study are "insiders" with clearances, they have seen only redacted material, or (most likely) both of those are true. All of those conditions ask me to believe non-public sources. I won't do that.

[EDIT: I hesitate to join the ranks of those who complain about shitty-but-common HN habits, but assuming for no reason that anyone didn't read TFA seems nearly as common as it is shitty.]

> I would encourage you to read the article.

I would encourage you to assume that those commenting on an article have read it.

What does your comment have to do with the article?

TFA is like most popular-media accounts of the "intelligence community": adulatory pablum divorced from widely-known facts, because secrets. If it's repeated often enough then simpletons with short memories will believe it. I'm not tabula rasa; I've actually been paying attention for nearly half my life. The world is simply not as dangerous as they've long struggled to convince us. Much of the danger that is real, was created by them. No thinking person can claim that the horrible things we've done over the last 14 years have made the world a better place. All that shit was supported by "intelligence".

It's even more obnoxious that this tiny study of non-public data is described as "even more striking because of its contrast with a famous earlier finding", which earlier finding was a vastly longer, more rigorous, and more scholarly study of more reliable, public data. That is, Tetlock's was a study one might call scientific. In TFA however it gets second billing to this spurious unrepeatable nonsense, because "professional analysts are more cautious than your average pundit". Sure, pull the other one.

Reply on news.ycombinator.com