It should be pretty basic to catch someone that's resetting user's passwords.
1] Do you log every single thing that happens on the machine?
2] If not, did you actually log the proper set of commands that were used to possibly commit a crime?
3] Granted you are not a small startup, do you have a process in place already to data mine, extract and analyze these potentially dangerous commands from log sets that are potentially TB in size?
4] Do you have the personnel in place with the qualifications and time each day/week/month to audit these potentially large results?
5] If you've caught something, aren't you already too late? Do you have some type of magical software in place that will somehow recognize these types of crimes in progress and cut off access in realtime?
Many of these articles totally ignore how these real world systems work (mainly I'm guessing because the authors have never been involved with companies that run 10k-200k servers around the world). It really is more complex than you think. Most companies that aren't startup size are continually playing catch up. Sure they need to put logging in place, auditing, yada yada. That is being balanced with the other day to day pressing tasks also. It always seems to be a game of catch up...
Do you have infrastructure in place to store all of that? How are you storing all of this content coherently so that you can go back and audit records of systems at a point in time you are concerned about? Do you have an internal Hadoop cluster configured (not trivial) and the hardware to ingest and process these logs? How are you actually logging everything? Do you have dedicated and qualified people to write this software and maintain it? When someone has su -'d to root, how are you logging everything they type coherently? Is your regex smart enough to ingest and contextualize multiline commands as root and add them to your report? What if no actual "bad" commands on your list were typed as root? What if I as a luser copied the "bad" commands to /tmp/kittens.txt, then su -'d to root and ran them there? Is this somehow captured? Are you 100% sure syslog is running 100% of the time on all hosts you are concerned about? Can't I as a sysadmin kill the syslog pid before I commit a crime? Are you using redundant syslog hosts? Are you using TCP and not the default UDP so that syslog doesn't drop packets under load?
Like I said, this isn't as simple as everyone thinks...
So I don't think your "do you have the infrastructure" argument holds too much water here.
The "do you have internal Hadoop clusters" question falls the same way. Sure, I don't have that connected to my 8TB of external USB drives plugged into my media server - but if I worked at Google or FaceBook or Twitter, I'd fully expect to be able to provision and spin up adequately sized clusters of VMs and storage to effectively consume and run reports on data that size and bigger. And it's surely not just the NSA and Google/Facebook/Twitter routinely dealing with collecting and processing data at that scale - any decent sized telco, any non-trivial web analytics service, any large financial institution, every HFT business, most bio-med businesses, probably every physics and astronomy department at any university – there must be tens of thousands of businesses routinely dealing with that sort of sized data sets.
It's not simple - certainly not simple enough for me to do it on a Mac Mini and a bunch of external hard drives – but I also don't think it's anything like "uncharted waters" territory. (I'm pretty sure I could find the expertise required in my 1st level LinkedIn connections, and have absolutely no doubt I'd be able to manage designing, developing and deploying exactly such a system if someone came to me with a high six or low seven figure budget.)
I also agree with the parent. There are so many possible scenarios and things to log that eventually you're playing a "logging" version of whack-a-mole. Even just managing these files (as the parent talks about) is really no trivial task. Honestly, I wouldn't even know how to begin managing a petabyte worth of daily data.
Second, I don't think anyone thinks it's totally trivial. But, this is the NSA we're discussing. Supposedly, their secrets are so special, the USA will collapse or something if they're compromised. If any organization in the world is set to handle an admin resetting people's passwords, it's the NSA.
Are you really arguing that it's just too hard for the NSA to notice a massive violation of policy?
The people you trust to admin systems are going to easily be able to abuse their power and it is very hard to stop them from doing so, without making their job so cumbersome for it to be near impossible.
[1] http://www.xceedium.com/solutions/privileged-identity-manage...
1+2) You could catch a decent portion of this type of behaviour with a tiny fraction of the effort. Logging local activity on individual user desktops is a waste of time, for example. Raising the difficulty of doing things undetected raises the bar significantly, particularly if users don't know exactly what's going to trip an alarm somewhere and get them caught.
3+4) Considering the nature of their work, I'd expect the NSA to be the right people for this with existing processes, suitable personnel and the storage/processing power required. It's their job to intercept and analyse large quantities of network traffic.
5) There was a period of around a year end-to-end in Snowden's case, and was in contact with The Guardian months before leaving the US (http://www.reuters.com/article/2013/08/15/us-usa-security-sn...).
Yes, it's a difficult problem (and impossible to comprehensively solve, doubly so when you don't trust administrator-level users), but when you're dealing with classified information in government, security is more important than in a regular commercial context.
If anything, the lack of decent internal auditing at the NSA is probably intentional.
I was required to manually go through each 24-hour output with a highlighter and look for attacks....
This was not a fun part of my job.
(Having sid that, I wonder how many people have patents or patent applications in for that idea right now?)