Citibank IT guy deliberately wiped routers, shut down 90% of firm’s US networks
tripwire.com
tripwire.com
I like to think that even though people see sysadmins are grumpy rage-machines in a basement most are super-decent people caring deeply for the systems under their control. Perhaps I've got a too rosy view of my fellow engineers.
I'd be very curious to hear what went on there on the people side of things. What led to this person feeling so angry that they would take an action like this. I'm also curious how it could even get so far. I have the benefit of being able to regularly talk to my manager so even if we did performance reviews in that way they wouldn't come as a surprise.
I'd be interested to hear it too, but in the grand scheme of things, it doesn't matter one iota.
A healthy work environment is as needed as compartmentalization and other schemes to manage abuse.
We do not know the extend and circumstances of the employee and his/her related story and the story of the company, so no judgement on that.
Someone could be a great team member, love their peers, and still react badly to the news they might be fired soon.
I don't think any amount of "people management" can fix someone who is already a bad egg, "configured" to react poorly to bad news. Some people will be out for revenge no matter what.
The more important thing to analyze here might be the technical measures in place to respond to configuration files being wiped out. Sure, an "insider" could maliciously wipe out the files... but couldn't a technical glitch just as easily do the same? The important lesson from this should be including adequate detection measures to immediately know when configuration files have been changed outside of the normal build process, and the ability to rollback when changes are detected.
One person should not be able to do any more damage than a technical glitch.
If he left his review thinking "they're going to fire me" then there was a miscommunication somewhere along the line.
There's a reason employers don't say "We're firing you at the end of the week... so uh... play nice until then, k?"
It's important to study that in these cases. I was convicted in a similar case and am actually a chapter in a CERT book, but they never reached out to me for my input, so they're completely oblivious to my motives and missing key facts about the case. I'd write more about it or speak at conferences but I've been able to bury that past and move on. Maybe some day.
I think he had a similar case to mine where he felt he was stuck in a situation with his manager(s) and felt like he had no recourse. HR should be involved in employee reviews and should provide a way for the employee to give feedback on his own review.
Another aspect of it is education about the law. I see case after case where the defendant had no idea he could face federal charges at all, much less one that can result in such stiff penalties. (The penalties he was facing were much, much worse if he had been tried and convicted rather than taking this plea deal.) That could easily be part of any degree program and/or employee orientation.
http://www.infoworld.com/article/2653004/misadventures/why-s...
Still no excuse to take down 90% of a company's connectivity.
I know it's a really out there example, but there are times where outright rebellion, even destructively, is morally acceptable. I don't know the specifics of this example, however.
Destroying a company damages all of that companies customers and employees as well. The difference between a surgical strike of a "bad guy" and carpet bombing a country.
Also, blowing the whistle can mean many things... in this case, it was a temporary disruption, not much different than an organized strike would be.
Yeah, isn't that what I said?
One issue I see here is that HR usually sides with management and tend to label you as a troublemaker whenever you disagree with something or someone. HR should always be neutral and they are usually far from it.
"They was firing me. I just beat them to it. Nothing personal, the upper management need to see what they guys on the floor is capable of doing when they keep getting mistreated. I took one for the team. Sorry if I made my peers look bad, but sometimes it take something like what I did to wake the upper management up.”
However, you don't go from happy trotting elf in candy rainbow land to martyr by the flip of a switch (at least not that I know of). So regardless I'd still want to understand what contributed to creating an environment that caused someone to retaliate like this.
What this guy - and Snowden - show is that today's IT professionals have tremendous power that generally they never choose to exercise.
If an IT shop in a major corporation decided to unionize and turn on its management, they would have the company totally by the balls. So many people in an IT shop are positioned to control so much, and management often treats its IT staff like disposable widgets.
It would take two+ hands to count the number of major multinationals that I could have seriously damaged singlehandedly in a matter of hours if I had ever gone rogue back in the day. Few non-IT rank-and-file employees have such power.
Of course the fact that I never did such a thing is one of the reasons I remain a well-paid consultant (trustworthiness and honesty is a consultant's #1 asset), but the fact is that many corporations are quite blind to the level of risk they undertake when they mistreat their IT staff.
If you have achieved a genuinely 100% ops-free setup, sure. Otherwise, an IT walkout will shut things down just as surely as a factory walkout.
I don't think that's really true in any meaningful sense:
In a growth company (e.g., a VC-funded startup), where most of the technical staff is focused on new development, a substantially reduced skeleton crew can keep the lights on, but "keeping the lights on" isn't going to keep such a company a going concern.
In a software company (mature or otherwise), the technical staff (other than internal IT support) are "factory workers", with the same situation as you describe for an iron factory.
In a mature, non-software company, a very large portion of the IT staff resources are devoted to just keeping the existing systems running; if they could do that with 10% of the staff, they'd already have cut the staff down so that was no longer true.
The same is true of a software company without developers has a vastly lowered burn rate. They need people for critical bug fixes, but a <90 day strike is recoverable and people are replaceable on longer time scales. Microsoft on a 90 day strike may actually be more profitable that year. It would be bad long term, but they don't need to keep pumping oil to pay for loans.
A mature company tends to spend a lot of time dealing with software rot. Using an older phone system for another 3 months is just not a major issue. Not having email is huge problem but that's the kind of thing a tiny staff can keep running. Unless they are at the point where they are already operating like this, but replacing a tiny team that walks out is not a strike it's a cost of doing business.
Most IT guys can do that.
If you "accidentally" forget to document that...
1) Nowadays almost everything depends on IT.
2) Majority of people still perceive IT as a "black magic". Arcane knowledge, something they don't understand, but depend on.
The reason why this doesn't happen is because it's an exceedingly dumb crime and most people are not sociopaths. Plenty of people like to fantasize about scenarios like this but it basically takes a terrorist mindset to actually go through with it.
In regards to the integrity of their employee evaluation process, I have an anecdote to share. We used a 360 evaluation process. My review set was my direct manager, plus 5 other people with whom I directly worked. That year, I received a 5 out of 5, the highest rating possible.
During the official manager / employee review sign off meeting, my manager noticed that my 5 was no longer a 5, but a 3. He canceled the meeting to investigate what happened. Apparently, his manager with collusion with a senior HR manager, went into the employee review system and manually overwrote the ratings provided by my manager and colleagues. The reasoning provided was that there were too many high ratings within my employee pool and I was selected by "someone" to be downgraded.
I can see going into a Citi employee review meeting, coming out, and wanting to nuke the entire site from orbit.
Where my friend works, it's more like yours and a select few get the stick. But they have this feature where you get reviewed formally if you have 3 consecutive "bad ratings", so mysteriously, no one ever gets the stick if they got it last cycle.
Just, for both of us, it's company-wide knowledge.
Most of us have the keys to the kingdom in one way or another and while I (and others) could never do this it doesn't mean I don't have sympathy for someone who does - for some justifiable reason. I can't imagine any that are justifiable... but I can ask.
I found the other linked articles about, "The Malicious Insider", a bit fucked in the head. Unapproved hardware! Email misuse! Unapproved workaround! Unapproved software! Are you fucking kidding me? That's a pretty far cry from this guy.
“They was firing me. I just beat them to it. Nothing personal, the upper management need to see what they guys on the floor is capable of doing when they keep getting mistreated. I took one for the team.”
“Sorry if I made my peers look bad, but sometimes it take something like what I did to wake the upper management up.”
Literally the "keys to the kingdom".
Sometimes the default level of trust is just really high...
Not having a great command of the English language doesn't seem to be a huge barrier to technical roles that are fairly insulated from the business side of things in enterprise environments, IMO.
On someone's last day they ran rm -rf / to wipe their desktop machine. I can't remember whether they typed it into the wrong terminal window or had mounted an external filesystem somewhere deep under their home directory and forgot about it, it was something like that. They wound up wiping some unfathomably large amount of data (I think most of a data center) before noticing what they'd done.
There were backups, but it was still a huge hassle.
Always a fear of mine. I always try to pause for a second before I run a command like that.
Because they now fear employees and contractors more than external threats it takes 30 tickets to different groups to set up a server. You can't chown a file in a directory you own without a ticket. You can use one of two old text editors. No new IDE's or modern text editors. You can't upgrade language runtimes without corp. approval. It's nuts. Ironically it leads to more attack vectors and it's ground their software dev to a halt.
It's simply an overreaction to risk. Just like insisting the company doesn't treat its employees like human beings is also an overreaction (and gross oversimplification of what is actually a complicated topic).
There were things like managers getting punched, ppl screaming etc but no malicious hacking that I'm aware of. Somehow one dude (I'm assuming :)) managed to take a dump in the elevator on the way out :)
They had EMS parked outside too. Wasn't pretty.
Someone set up a digital timebomb on the systems. They were one of the people let go, and a few weeks later, things crashed all over the place.
One amusing bit - a few months later, another digital time bomb went off. He had hidden another one that they had not found, even knowing of the first one.
Despite laying off about 10% of the staff, management had a good idea of who it was, but they had been clever so it was difficult to get evidence it was them.
However, potential to do damage usually supersedes potential to benefit the business in the notice period. A company with sane HR has preventative policies in place:
- Never have single-person key-man risk in the first place; always have a backup/deputy designated well in advance because anyone can be hit by a bus. Helps with covering holidays too.
- For highly sensitive jobs like sysops or people dealing with customer money, revoke access to systems first thing in the morning, then tell them, then immediately escort them out.
- Don't have shared or master passwords anywhere, but have user-based ACL instead so access can be revoked.
This leads to a messy set of compensations:
* When we fire people, they may take revenge, so we escort them out immediately
* ...but then people start assuming the worst when we tell them they are doing poorly, so we stop doing that and just fire them once it's bad enough (escorting them out immediately at that point)
* ...so then they start trying to guess when we are unhappy with them, and now we have employees that weren't at any risk of firing taking preemptive revenge...
It's a mess, and the less respectfully you treat people in the name of "policy", the more it can escalate things.
> “Sorry if I made my peers look bad, but sometimes it take something like what I did to wake the upper management up.”
Ah yes, the old double 'they are harming me by holding me accountable' and 'I am helping them by holding them accountable' defense.
What I'd like to know is how mindsets like this are formed in life, and how to prevent them. One's employer doesn't owe one a job; one owes one's employer fair labour in return for one's wages. The world does not consist of shadowy forces plotting against one. Don't minimise one's own agency: be an active force for good, not a passive subject of whatever happens.
That kind of thinking is exactly what breeds this kind of stuff, though.
Labor laws disagree to a certain degree -- at least in that you can't fire someone for any reason you choose (see the EEOC poster in your breakroom/mailroom/kitchen for examples). Also employment contracts can stipulate various things that prevent a company from firing you, particularly in union jobs.
I've found going by the motto "don't be a dick" tends to be a decent deterrent against doing something stupid like maliciously wiping company data and getting sued and/or going to jail for it. If it's your employer doing it to you, there is plenty enough opportunity in IT to find work at a place that doesn't have a hostile work environment.
If I had any advice it would simply be to avoid letting a corporate environment allow you to forget you have real people as employees with feelings. "It's just business" doesn't translate to everyone.
What does this mean? Do you mean "provoked"?
Note, I am aware of the use of trigger as in "trigger warning" for potentially disturbing content. Even with that context I'm lost. Did you mean "literal" literally?
The only way to rid yourself of the threat posed by other humans is to run a single-person company, and even then you might screw yourself over.
You will never be able to completely neuter a threat that does any useful service for your company, and it's foolish to think you can. Any useful amount of delegated power or responsibility without the ability to misuse it is very rare, if not impossible.
There are people unable to take even the nicest, most constructive of criticisms (or even perceive non-critical statements as such). Hard to say without knowing the people involved on a personal level.
There are analogies to broader threats, say, to entire countries, and these days a lot of people are more interested in being antagonistic and hostile in dealing with the threats than working to understand them and their grievances. The two sides may never agree, but the effort makes all the difference.
I argue that these dynamics are relevant from a criminal psychology and a government-hacking standpoint. Being mostly derived from firsthand resources, Toffler, Bernays, et. al, along with some authors whose mention is straight-up dangerous, I am constantly seeking additional resources.
Uh. Hack the planet.
Edit: or don't. U du u.
This is arguably what keeps shows like Mr. Robot on the air btw. We will tolerate hackers down certain pathways and within certain constraints. Even if they're "insane". Leave the pathway and God help you.
Sometimes staying on the pathway is dangerous too, but my job is to motivate for great justice.
Edit: Don't kill or torture anyone. This should be in your base operating code anyway; if it isn't, enforce it down to pith level.
The EFF has a good breakdown of weev's sentence and the D&D-like charts and calculations used to reach it. He got hammered for "special skill", "sophisticated means" and "means of identification" bonuses.
https://www.eff.org/deeplinks/2013/03/41-months-weev-underst...
https://en.wikipedia.org/wiki/United_States_v._Swartz
I wonder where they started the negotiation before settling on a 21 month plea deal
"facing 35 years in jail" is inaccurate in the sense that it would've been impossible for him to receive such a sentence.
"If convicted on these charges," said Ortiz, "Swartz faces up to 35 years in prison, to be followed by three years of supervised release, restitution, forfeiture and a fine of up to $1 million."
IIRC it was also precipitated by perf reviews or discipline.
Hope AWS has better systems in place to make something like this nearly impossible to carry out.
[1]http://m.sfgate.com/bayarea/article/S-F-officials-locked-out...
So why is the router attack any worse than a "Delete exchange server + all backups" command, or the "use ansible to remote format entire server farm in 1 click" command?
You have to trust employees at some level. If people know that doing things like this will equal jailtime, I would assume that would stop most people.
Now think if this employee lived in Russia and did this remote. What would the recourse be?
In Unix, yes, but there have been systems without a single ultimately-privileged user.
One could imagine a system in which ultimate authority belongs to a 51% share of stockholders, whose keys delegate authority to the board of directors, who delegate authority to the CEO or CTO, who delegates authority on down the line. Each certifying party could revoke (or allow to expire) authority prior to firing a delegee.
> If people know that doing things like this will equal jailtime, I would assume that would stop most people.
What I want to know is how people grow to physical adulthood without realising that this is wrong. One simply doesn't destroy others' property.
In the end, society only functions because the vast majority of people think they are better off following the norms set by that society. When they no longer think this because of emotional tunnel vision or a real lack of hope, whether it be real or imagined, or pervasive or lasting just long enough, their actions are no longer predictable as a rational member of society. Unfortunately, that means in some cases, it doesn't matter what the consequences are, there will still be the occasional incident.
Well, every deploy to production in any company is the result of a shareholder vote — it's just made obvious.
By using delegation and certificates, an employee who is delegated authority to deploy to production can do so without requiring a majority of shareholders to actually vote on that particular deploy.
That's pretty awesome.
> This sounds like a Utopian paradise not the realities of IT in 2016.
We've had the ability to do this for almost twenty years: it was made possible by RFCs 2692 & 2693 in 1999. The necessary processing could be performed in a split second.
Which is fine, but seems like a roundabout way to do it.
Take note - just in case, don't wear a 'hoody' as people may think you are a 'hacker'.
The thing is that that you can't do much against someone who has root access for a living.
There are other ways to make a point, especially when you're prepared to get fired. Damaging everyone else is selfish and stupid.
He did, however, hold a Rubix cube as a way for Greenwald and Poitras to identify him in an early meeting. http://www.cnet.com/news/edward-snowden-and-the-rubiks-cube-...!
In banks, significant decisions (like personally authorising multimillion dollar transfer) require two signatures (of two senior officers), so they can't be abused by just one person going rogue.
It can be imagined that we apply the same approach for system administration - "pair-programming" work only, perhaps with proximity cards which detect when one of the admins is actually not at the desk.
In contrast the banker has a more relaxing experience putting his signature on a document. It's stressful enough running root commands sometimes, and I can't imagine how horrible it would be to require a second person to be on-call/pair/double-check/etc. I would have long ago quit the profession, which would be impossible to do anything in. Also systems would take longer to fix, so organizations would be displeased.
Vet your sys admins correctly before hiring, keep an eye on people's stress levels, which in 2016 America apparently doesn't matter to anyone anymore in IT, and don't piss off the sys admin.
"Two-Person Control Administration: Preventing Administation Faults through Duplication" by Shaya Potter, Steve Bellovin, and Jason Nieh
https://www.usenix.org/conference/lisa-09/two-person-control...
The video isn't there for some reason, but the paper and audio of the presentation are.
basically don't allow people to commit changes directly to the underlying system, require the changes from 2 administrators to be captured and then compare them for equivalence. (what equivalence means can be bit for bit equality at the most basic binary level, or perhaps some parsing that can verify equivalence at language level for configurations)
if the changes are equivalent commit them. otherwise point out the differences.
As the paper also discusses, can be used with a single administrator to make the administration auditable, simply capture all changes before they are committed.
Inquiring minds want to know.
> Finally, Snowden’s physical location worked to his advantage. In a contractor’s office 5,000 miles and six time zones from headquarters, he was free from prying eyes. Much of his workday occurred after the masses at Ft. Meade had already gone home for dinner. Had he been in Maryland, someone who couldn’t audit his activities electronically still might have noticed his use of thumb drives.
http://www.nbcnews.com/news/other/how-snowden-did-it-f8C1100...
https://www.schneier.com/blog/archives/2013/08/nsa_increasin...
The worst part about it is that to be completely on the ball with internal threats, you need to be constantly paranoid, assuming by default that any employee with enough access is a threat, or a threat waiting to happen. Such things do not add to team morale or cohesion, but in this case, a pound of prevention is still better than a metric ton of cure.
Backups could be maintained on a regular schedule, including full router backups. Others in charge of this could be given create and append access to the backup systems, but not delete.
Once you delineate split rights and controls, then you can mitigate the severity of these attacks. Yes, douchebag-router admin can still zero out the routers. But you can be in business in less than a day with those configs logged.
Point in case, as parent commentator said, it's not the kind of problem that you solve with technology.
But thanks for assuming I'm some inhumane business exec, or a VC.
In the company I work at, we have something similar for database executions. In Dev you can run anything you want. However in prod, everyone only has read-only access. If you wish to execute something with higher authority, there is a web page we go to, where we can paste the code, a review board request, a ticket number (if available) a business authorizer, and a technical authorizor (other than yourself). It sounds like a lot, but it's usually pretty quick. Every command is then logged (which I believe is more for SOX reasons). It's annoying, but it's not that terrible of a system.
Of course a sane company wouldn't have a bus factor of one for such a process, but not all companies are sane.
At some point, you'll get somebody with full access to your system. You can reduce it into a very small window (at big cost), but you can't make it go away.
1. deploy the system
2. create new admin account with a long random password and/or 2FA physical token
3. print the password out, seal it in an envelope and put it in a safe not accessible to any IT person; place the 2FA token in a separate safe controlled by different person than the first safe;
4. remove all other admin accounts - with the other present technicians checking that no other account will have access
5. if needed, with proper authorisation you can retrieve the password, and implement the required changes (again, with a second person controlling what exactly is being done).
If the systems require changes rarely, then this cost isn't that high.
I assume one could refine the process by binding sets of commands or capabilites to specific cards.
Would also be rather handy in preventing remote access attacks.
I'm genuinely curious, is this the actual charge? What specific law did he break here?
Edit: He plead to I think a single count of the following 18:1030(a)(5)(A) and 18:1030(c)(4)(A)(i)(I) and (B)
http://www.leagle.com/decision/In%20FDCO%2020160411834/U.S.%...
https://www.law.cornell.edu/uscode/text/18/1030
It's a perfectly reasonable principle to write into law, a password or key or whatever doesn't give an employee the right to damage the systems of their employer.
There's interesting details behind this; were they really firing him? Did Citibank make the horrible error of telling an IT worker that they are fired but not immediately revoking their privileges? Could management have handled his performance issues better, or were there red flags that he should have been let go sooner?
All of that could be interesting but because this is just clickbait blog spam none of it is in the article.
I almost accepted an offer to work in their NY offices. When I was being interviewed, the Manger of the group was divulging his dislike for the person I was going to report to.
It's a sick place & the IT guy in the article sounded like he was being abused. He shouldn't have done what he did, He should have documented the stuff he went through and maybe get a lawyer and sue for pain and suffering.
I mean, there's very little you do about this particular scenario other than some small mitigations (at the end of the day you'll have to trust someone with something) but it's genuinely amazing how vulnerable the processes inside large, multi-million companies are to any internal attacker.
Maybe the scenario is too rare for companies to care, so when it happens they just accept the cost of once in a while instead of setting up additional security policies which will turn procedures cumbersome.
What stops bad person in finance from making a huge wire transfer to a hidden account? What stops the CEO from plundering the company? What stops the average driver from taking out a bunch of pedestrians? What stops a doctor from purposefully poisoning one of her patients? What stops a parent from abusing their children?
Only good will and care for the future. And any system in place to stop that stuff from ever happening would cost more than the benefit.
Obviously stupid on his part. He needed a patsy. He needed to get someone else to do this. Some kiss-ass golden boy whose only value to the company was unerring unthinking loyalty. I have seen these douchebags destroy so much infrastructure, even without a malicious shadow revenge plot.
If people from marketing, accounts, sales or wherever else are giving you problems then you can sometimes automate them out of existence. This is particularly so in ecommerce where the gig is with one of those regular small businesses that haven't quite realised that they need to be an internet company.
What you can do is automate all the accounting, doing it right with decent database schemas and everything the server has to offer. The accounts can then be a live thing and not some dead Excel export from some 90's era accounts package. Then that guy in accounts who spends 48 hours a week to get to those reports is redundant. He does not bother you any more and the managers are pleased they can see their charts online whenever they want without someone having to prepare them specially.
Meanwhile in customer service, let's automate all of those procedures and put the process in code. Put up some forms that work and make it so easy for everyone that the customer service team no longer need to be the size that they are.
Then there is the non technical web team, the layer of cruft that collects with non-web companies after the 'webmaster' in the corner of the room is deemed to need 'help'. These people are like Chinese whisperers (no offence, just turn of phrase, sadly...) in that they liaise between those making content 90's style in places like the graphics/print department and then mangle those assets for the web. Again, put them out of a job, put procedures in place so the guys in graphics are aware of this web thing and create content first and foremost for that. Simple things like file naming procedures can be all that is required and the guy that collects 'assets' from one part of the company to then upload them is no longer needed, neither are those pointless meetings that were needed.
Have a problem with the actual graphics department and their print media tendencies? Move the requirements on to responsive SVG instead of the legacy PSD/PDF exported to blurry jpeg they love. Automate all that processing of 'assets' with some imagemagick. Take all the design layout out of 'design' and into UX with everything 'aria' tagged.
Then there is HR. Put in place the tools for them to manage recruitment in a modern online way and get rid of the need to have people copying and pasting stuff out of Word/Excel/Email. Do that right and you can erase the need for there to be an H.R. assistant running around doing this stuff.
Distribution and warehouse stuff is also similar, why use those in-house legacy systems when you can go XML SOAP to the distribution center? Get rid of the heart of the existing company - the legacy IT systems - piece by piece and the people that go with it.
If you dislike someone and you can automate them out of existence then that is a very good incentive to get on with that aspect of the overall project. Having done this, the company is even more reliant on IT as there is now no Excel spreadsheet solution.
Note that the above is reverse empire building, it is doing the right thing and making a company a web company rather than a 90's SME built on Outlook and Excel. Also it is a different exercising of power to going for the jugular, 'rm -fr' stuff. You also don't have to strictly automate people out of existence, you could make their job massively easier with less repetition and difficult sums done and have it so that no managers know that a tedious hard job is now coastable, even making it so that managers don't have the shortcuts that the staff do.
A society where IT workers have great power over corporate fortunes but little political representation is going to be unstable...
That being said - I do agree with your sentiment that the more dependent society becomes on technology, the greater the risk posed by those who possess knowledge of how to operate and maintain that technology. This doesn't just apply to human beings too - electronic actors (e.g., automated systems or even AI) may someday if not already hold just as much power.
The question in my mind then is how to "democratize" this power, so to speak.