Yeah, North Korea was just waiting for the NYT reporters to give them up-to-date intel on why their launches were failing. That's really the only reporting Kim Jong Un trusts, so I hear.
What the whole Abdul Khan episode brought to light some years back was how Western agencies underestimated the technological and political skill and motivation of many so-called "third world" countries. Arguably that blind spot was caused by the deep-seated racism and cultural elitism of Western countries. There were plenty of opportunities to discover the illicit trade; we just weren't convinced they were capable of it, despite all the evidence to the contrary. Our leaders never took it seriously because their expectations were that these countries were still too backwards.
Yet another example of why we really need to face prejudice head on. We simply can't accurately assess the state of things if we're blinded by so many poor assumptions, particularly the ones girded by racist attitudes.
If American culture continues casting Asians as uncreative copycats; Muslims as radical fundamentalists; Africans as irredeemably corrupt; and South Americans as hopelessly socialist; then we're in for decades of more wild and very much unwelcome surprise.
But priorities are set by the elites. We rarely encounter "slam dunks" that force leaders' hands. And so our strategic direction relies heavily on the higher-ups making decisions that are still highly susceptible to bias. Heck, we elect leaders precisely because of their bias--Obama because we thought he had an inside-line on the Muslim thing, Trump because we hate Muslims. I exaggerate but it's not far from the truth.
That's why popular culture can be so critical to foreign policy. Two decades into peak globalization, there's no way to hide these things. This isn't the Cold War. Leaders can't do the smart thing while paying lip service to the electorates' prejudices. Even if we could stop all the leaks in the U.S. government, you can't hide the consequences of policy.
Now, maybe the intelligence community came around much earlier. But certainly Congress was still having trouble coming to terms with the reality. Why? Because it was just so difficult to believe (and still difficult to believe) that such a repressive society could achieve such a feat.
It is, admittedly, difficult to believe for me. But I understand part of that difficulty is because Westerner's believe that such feats aren't possible without a culture of individualism that unleashes individual initiative. Maybe such a culture is superior. But it clearly isn't necessary. And that should make us question much about what we think other countries are capable of, and when they'll be capable of it.
194x-195x USSR wasn't a pinnacle of democracy and freedom. And technology and science back then was much less advanced than today.
I think that's an exaggeration. I'd note this 2013 NYTimes article[1], which notes that all US intelligence agencies agreed that North Korea tested a bomb in 2006, but were unable to come to a consensus view as to if they were capable of miniaturizing it enough to put on a missile:
an assessment by the Defense Intelligence Agency that it has “moderate confidence” that North Korea has the ability to shrink a nuclear weapon and fit it into a missile warhead surfaced at a Congressional hearing. That conclusion was disputed by James R. Clapper Jr., the director of national intelligence, who issued a statement later in the day saying that it did not reflect “the consensus” of the nation’s intelligence community.
[1] http://www.nytimes.com/2013/04/13/world/asia/contrasting-vie...
The CIA will of course issue the standard Glomar response (or simply remain silent) and NK will have no more or less concrete information than they had before.