Disclaimer: I've conducted personal research on politics and gov between 2003-2014; I've been exposed to non-public facts and discussions the likes of WikiLeaks as we have come to know it since; however I do _NOT_ have "insider knowledge" of any kind regarding these issues. I've merely learned to read between the lines, so to speak (read: Snowden speaks in a fairly intelligible language to me, like many others privy to how gov works I'm rarely wrong about my assumptions in the long run; and believe me when I say it's easier than one thinks: we tend to overcomplicate matters as adults, obfuscate ourselves the underlying 'whys'). At best, these are educated guesses from historical and modern facts, which represent my _opinions_ and only mine. Also, by far and large, I do _not_ subscribe to conspiracy theories. More like the opposite: I do advocate that govs run pretty much as they should and near the best of their ability most of the time. The real question is indeed less about intent and more about capability (in a broad sense, including intellectual/philosophical 'enlightenment').
With this out of the way, here's the TL;DR: like most corporations executives, the fundamental goal of most officials in place isn't to conduct their mission/mandate but to further their own power; two motivations which ideally go together but in actual fact seldom do. Only the illusion of the former is actually necessary to achieve the latter.
I won't elaborate much because this view amounts to a borderline revolutionary thesis of sorts (calling for a new kind of regime, in place of the disingenuously called "representative democracy"). It's interesting but it's a book, not a comment.
Just recall history and ask questions about:
- The subtle and yet vast spectrum of 'corruption': where does it begins, how far it goes, even after every involved party has died?
- The illusion of safety or 'acceptable' conduct that officials (or execs) can convince themselves of, regarding their organization and themselves: how cognitively biased can human beings be, paradoxically increasingly so as the stakes go higher?
- When we could predict with reasonable confidence that things could go very wrong very fast, did we always manage to prevent just that from happening? Did we even try to the best of our then-capabilities? (Don't even go Godwin on that, not even wars, think market crash or viruses).
- How often in history do we need, as whole societies, the lessons of examples and dire consequences before we put our act together? What is the limbic value of 'could' versus 'hasn't yet'?
Once you realize, as I did some time ago, that we are much less rational beings than anything else, every mind-blowing decision you've witnessed in your own life by people who should have known better becomes not just possible but likely and even understandable, if not outright predictable.
My personal contention is that we'll basically need to go through a 'black swan' of sorts to really deeply learn about electronic safety; much like knowing about germs and viruses did not automatically provoked decent health procedures even by health professionals, let alone the public. But now we do wash hands and sanitize rooms. Just consider how hard a lesson it's been to learn for humanity.
We are in the infancy of the electronic age; I'm afraid it will get worse before it gets better. I just hope it will take the form of a temporally 'short' catastrophy rather than a whole new medieval age of sorts, "dark ages" as seen from a hypothetical future; but the more I observe human beings today, the less confident I feel about us taking the high road directly. Especially when lives are threatened not clinically but rather socially and psychologically. Our survival instincts tend to be numbed when life/reproduction isn't directly threatened.
At this point you may go Orwellian because it's almost less frightening than what's potentially facing some of us on this planet, and our societies appear to be both very resilient yet very susceptible to flaws (probably goes hand in hand). And when there's a flaw... be it legal, psychological, or otherwise... you know someone someday will exploit it. Circumstancial aspects like this ME merely serve as paths to what will eventually become history, but alas it's almost impossible to predict which particular aspect will be the trigger, as impossible as it is to shield ourselves from all angles. But whatever, it too, shall pass.
This isn't to say we can't mitigate because we absolutely can and should (I do try personally), but we can't really alter the shape of things to come because it would amount to changing human nature before the fact. Never happened in history. I don't think evolution works like that, period. Sorry for taking the discussion into a rather philosophical light, but that is my best answer to the 'whys' you asked and seem concerned about —concerns that I share but fail to astonish me nowadays.