Anonymous speaks: the inside story of the HBGary hack
arstechnica.com
arstechnica.com
It is easy to look at almost any intrusion and attribute it to poor defenses. If HBGary didn't have a SQL injection, they'd have had a XSS vuln. Or a employee would get spearphished. Or an attacker at a local coffee shop would compromise a mobile client. Or a backup service would get compromised and unencrypted. Or a interviewee could plant a network listening device. Or the CEO's daughter could win a pre-owned iPhone. Or a secretary gives out a VPN login. And so on and so on.
Did HBGary suck worse than usual? Possibly - but consider Google china got hit by ie6+acrobat vulns, DOD lost hundreds of thousands of classified documents from an air gapped and physically controlled system to a private, Open BSD may have included side channel backdoors, Kaspersky lost their source code, PS3/iPhone/Xbox/HTC etc. are unable to secure their platforms.
The truth is, a motivated attacker will rarely fail. Anyone reading this would be unlikely to survive 24 hours of a coordinated attack whether it's done by 16 year olds, chinese university students, russian mafia, FBI or simply nerds that know how to google vulnerabilities.
Fighting back against a group like Anonymous provides the same asymmetric warfare problems as the US military experiences in fighting terrorists, including the inability to respond with similar tactics for legal reasons.
Bottom line is, almost any organization can be subject to this kind of embarrassment without warning.
They failed in:
- Keeping their systems patched and up-to-date.
- Convincing/forcing their users to use strong passwords.
- Convincing/forcing their users to use separate passwords per system.
- Convincing/forcing their power users/admins to use a unique, strong password on key systems (i.e. Google Apps Admin).
- Not-invented-here syndrome (or maybe security through obscurity -- hey! if we use an obscure CMS then it won't be exploitable!) with respect to their CMS. I can be a little lax on them here. Had they chosen 3rd-party software people would doubtless be railing on them for which off-the-shelf 3rd party software they were using (e.g. had they been exploited through a Wordpress vuln, then people would be lambasting them for using Wordpress vs <insert-cms-here>).
You can take almost any intrusion and write it up in wildly different ways.
If HBGary had not failed in everything that you listed, odds are you would be listing some other comparable set of failures:
- something somewhere is always unpatched and out of date
- humans always deviate from best practices
- 99.99% of intrusions involve traditional threats, well known vulnerabilities, unpatched systems and human error
I could write up 100 different intrusions done in 100 different ways and almost always make the victim sound incompetent, or like a real life spy novel, or make the defenses sound like fort knox, or make the intruders sound like gods, or make it sound like my product would have prevented them, or draw the conclusion that the security environment is hopeless and out of control.
In the end it doesn't really matter how it happened or what I make it sound like.
Bottom line: Did you get owned [Y/N]
Really, you can go years without patching if you choose your software properly. (Except browsers - those just suck.)
[1] e.g. https://github.com/mojombo/jekyll [2] e.g. http://www.openbsd.org [3] e.g. http://www.openwall.com/passwdqc/ [4] http://www.openbsd.org/faq/faq14.html#RAID or the equivalent for other OSes [5] All over the internet. Or set up a Kerberos environment and get single-sign on too.
As I've mentioned elsewhere, competent security needs to take into account sociological/economic analysis. Only looking at the technical and organizational side is literally just playing with yourself.
For example: if you're a major technology company, taking the step of publishing wildly popular content with DRM means you're going to be taking on a lot of highly motivated opponents. History demonstrates that this isn't a fight that you want to take on. Now consider: if you're a security company, what do you think is going to happen when you take on a subset of /b/?
Here's a hint: before you're in the position where you're risking the ire of a large, technically savvy population that's had demonstrated success taking on other corporations with comparable or greater resources than yourself and a history of flaunting the law, it really behooves you to do some preparation.
If you've been saving some chump change by keeping your mailserver on the same machine as your webserver, and you're about to take on /b/, now's the time to do something about it.
That's like some athlete not checking if his shoes are laced up properly before the event.
Did this security company ever audit its own security? Either they didn't or they did an incompetent job of that. Would you trust a security company that doesn't eat it's own cooking?
The other side of the coin is that attackers don't publicize their losses. Every attacker has a limit to their skillset. If they can't compromise someone they won't announce to the world their failure, they'll just pretend like nothing happened and move on.
I have to disagree that this one is really a best practice at all. I have dozens of different accounts on different computer systems, if I didn't do at least some password reuse I would have a hard time writing them all down, and remembering them would be totally impossible.
With that said, I absolutely think passwords on critical systems must be unique and strong. But I have maybe three systems I consider critical and dozens that not. My bank password for instance is unique and long. But I am more worried about being able to remember my passwords than whether or not someone who gets into the account I use to play Go online can also get into the account I use to play chess online.
There are a variety of good password vault programs out there. I keep my passwords in a KeePass 1 file on Dropbox. I can run Keepass on Windows, Linux, OS X, and my iPhone and iPad. There are a variety of low-cost and free options that are about this good or better.
Sometime, I'll be working on a Keepass compatible iPhone program that can access Dropbox directly.
Yes, it's a pain in the ass (at present), but yes, you should be using different passwords for everything, and pubkey authentication where possible.
The problem of maintaining an encrypted master password list for many different accounts is just a technical one. It will be solved. Keyring managers already do this. I noticed the latest Chrome linux builds use the desktop keyring manager now for saved passwords, rather than storing them unencrypted in the browser's password store.
Personally, until these keyring managers are mature enough, I use a few simple scripts: one which generates a new semi-pronounceable password with random chars, one that adds a new account to a gpg-encrypted master password file, and one that queries the gpg-encrypted master password file when I've forgotten a password to an account.
Though, as you alude, this will become easier as keyring managers mature. I am also hoping that as security technology matures more places will move to forms for two factor authentication.
Which systems were these? I didn't see anything that implied they were compromised through a missing patch.
If you're referring to the CMS, then that could just be a bit of custom code. We don't know.
"The only way they can have some fun is to elevate privileges through exploiting a privilege escalation vulnerability. These crop up from time to time and generally exploit flaws in the operating system kernel or its system libraries to trick it into giving the user more access to the system than should be allowed. By a stroke of luck, the HBGary system was vulnerable to just such a flaw. The error was published in October last year, conveniently with a full, working exploit. By November, most distributions had patches available, and there was no good reason to be running the exploitable code in February 2011."
When an attacker uses rudimentary techniques that have been well known for many years and have straightforward and low-cost counter-measures, then you should rightfully be disgraced. More so if you are a security company.
It's not as though the attack against HBGary was like some expert safe-cracker routine. Rather, it was more similar to someone walking up to the front door, finding it locked, then finding a key under the doormat and letting themselves inside. There's no excuse for that. Not if you have any sort of obligation to maintain a level of security and secrecy.
That being said, there's absolutely no excuse for that sort of slap dash engineering today. It's dead simple, even in PHP, to use input sanitation, or to use parameter binding / prepared statements to avoid SQL injection vulnerabilities. Those sorts of best-practices have been well known for at least the last half decade.
When you call yourself a "security company" then it is not too much asked to please not expose a half-baked PHP Application to the public. It is not too much asked to have your team adhere to the most basic password practices.
A defender needs to constantly monitor, test, review isolate and basically never make any mistakes.
This is bordering on FUD.
No, as far as your network presence is concerned there is a very finite number of attack vectors. For most companies there is no reason to expose more than a very small set of services to the world. Hardening these services is well understood.
If I only open Port 22 and 80 to you, and the webserver will serve only static files, then you'll have a pretty damn hard time owning that box, unless you have access to very rare and precious remote exploits for the kernel, OpenSSH or nginx. And unless I make very basic mistakes in configuring these things.
Moreover good security is layered. It's absolutely ridiculous to try to come up with excuses for a security company having their CMS broken into and that being enough to effectively travel their entire network.
Any admin worth their salt will put the company wordpress on a separate server, with zero trust-relationship to the rest of the infrastructure. It's a no-brainer.
Yes, incompetence is widespread. But please call it out for what it is and don't try to come up with justifications.
That is something the IRA used to say, they only needed to get lucky once, whereas the police needed to get lucky all the time. Of course humiliating someone on the Internet is a world away from blowing up a shopping centre. If the consequences were more serious than embarrassment, then a lot more resource would go into guarding against it. Schneier talks about attack trees (http://www.schneier.com/paper-attacktrees-ddj-ft.html) - always look for the cheapest vulnerability.
Incidentally there is one online group who could eat Anonymous for breakfast - Mumsnet. If Anonymous ever took them on, they'd be grounded before you knew it.
> Incidentally there is one online group who could eat Anonymous
> for breakfast - Mumsnet. If Anonymous ever took them on, they'd
> be grounded before you knew it.
Any journalists out there, this is how it's done.
"Even with the flawed usage of MD5, HBGary could have been safe..."
They homebrewed their own password system. Can someone switch on the tptacek bat-signal?
the story says hbgary hired an outside company to make this cms for them, which may explain the crappy security on that particular system.
Can someone switch on the tptacek bat-signal?
thomas' security company also got hacked a couple years ago and had sensitive information plastered all over a mailing list. rumor was that it happened via their use of wordpress for their weblog.
i guess the moral of the story is... you will get hacked by crappy third-party software?
Doesn't that make them look even more amateurish and incompetent? They chose an insecure content management system and, most importantly, they didn't isolate it enough. So penetrating that resulted in a complete penetration of their site.
If they were selling hand-made baskets, nobody would blame them, but they sell "security" and charge big bucks for it, so they deserve the ridicule.
It is an interesting perspective I guess on selling "security", both as a service and a product. One can charge lots of money, but unless there is a serious attack and penetration, it is hard to know what the quality of they security product is. Of course once the penetration happened, there is at best pity and at worst ridicule and blame.
> If they were selling hand-made baskets, nobody would blame them, but they sell "security" and charge big bucks for it, so they deserve the ridicule.
I disagree. Anonymous were a highly motivated persistent attacker. It doesn't matter whether or not there was SQL injection involved, they'd just keep on going until they get in regardless. If there wasn't a SQL injection bug there'd be something else. Tptacek's company has been hacked into, our website got hacked into years ago (through having shared hosting - someone else had a SQL injection bug on the same box and the hackers defaced every site on the box. The difference is that we did a risk analysis beforehand and decided to never to store sensitive data there nor use the same credentials for that account anywhere else). Given a long enough timeline, everyone gets hacked. While the SQL injection bug was the way in, the real schoolboy error was Aaron Barr using a weak shared password for Google Apps admin.
I've been reading through some of this HBGary stuff, and I have come to the conclusion that Aaron Barr is kinda a dipshit.
Read the email analysis at http://www.wired.com/threatlevel/2011/02/spy/ and its filled with Aaron Barr "hacking" into people's facebook accounts and then posting pictures of their kids as if he made some awesome discovery.
And that's precisely the difference everyone should look for when hiring a security company.
No more than google choosing a linux kernel with a privilege escalation bug for Android, anyone using OS X in 2009 while a remote jdk bug sat open for 6 months, anyone using windows+ie in dec '10 or jan '11.
Unless you can explain how to only buy software that will never have any vulnerabilities.
This wasn't quite like Google choosing a linux kernel with a priv escalation bug or Apple leaving the JDK unpatched for 6 months. This was more like Google missing a great acquisition opportunity because they couldn't find the relevant documents on their internal fileserver, or Apple's website only rendering correctly in IE 5 because that's what they were using to test it.
They specialize in thinking up new ways to attack OTHERS, using OTHER peoples' tools. It's a huge problem in DC. A bunch of people telling other people what to do, without little idea or experience how to do it themselves.
Security through obscurity doesn't work.
As soon as someone who knows what they're doing comes along, you're in trouble.
Security through obscurity isn't a replacement for other strategies. That doesn't mean that it's useless; just that if people are relying solely on it, then you can pretty much bet that they're screwed.
Getting employees or users not to reuse passwords is probably the hardest thing to do.
Also, Ars' coverage of this story has been great.
For most, its like flossing every day. You know you should... but do you?
Also, changing keys isn't that hard. You just re-run ssh-keygen and delete the old key from authorized_keys and replace it with the new one.
The truth is passwords are not like toothbrushes - they don't actually need replacing every three months. Only if some event has occurred (e.g. a sysadmin leaving the company). They don't get weaker over time. Why not let someone keep the same decently strong password for as long as they're an employee? I guarantee this will be actually more secure.
Passwords do get weaker all the time, to the extent that they are used in multiple places. Changing the password on different systems on different schedules discourages password reuse. It also means the 'active' password is much less likely to be the password the employee used on a random news site they logged into once to comment.
There is obviously a balance to be had, because frequent rotations may encourage people to choose weaker passwords, but there is certainly value in expiring passwords.
But it doesn't, it really doesn't. It just results in people buttonholing sysadmins in the corridor asking "when are you going to stop dicking around and implement SSO?". Not long after that, people just start ignoring security advice altogether.
Where it does discourage reuse is across multiple systems, and with websites. Making me change my corporate password every 90 days is an effective way to ensure that I don't use my current password across a large number of websites. Maybe I'd go to the effort of making my gmail password the same as my corporate password. The password on that random news site account I forgot about? No ways.
It's not a panacea, but password expiry does effectively limit the spread of passwords in many cases.
By changing passwords every N months, you eliminated the ability of someone to crack the hashes and obtain a cleartext password where they previously only had a hash.
That time window has gotten absurdly short, however...and with Pass the Hash and MITM, I don't even need your password anymore :(
It's just worse because HBGary isn't eating their own dog food. Think about that.
As a security specialist myself, I always do my very best to follow best practices, not only to protect myself, but to show others that I am willing to follow my own advice. I have met security people doing presentations that need to elevate to admin, and they are logged in as admin already. Sometimes even with UAC turned off. This completelly shatters my confidence in them. I am always logged in as normal user and have long (20+) passwords, and make sure people see that I have long passwords. I don't hold others to the same standard, but I know people. If they see that I use 20+ characters, they will not think that the 10 that I want them to use is that bad.
This is the way the security landscape should work. We should all set ourselves to much higher standards than the advice we give others. We should always follow up on it. We KNOW lazyness will cause breaches, so therefore we should never be lazy when it comes to security. For a security company - and especially the president - to have such low security lowers the confidence for the whole industry.
Yes, security is asymmetric. That is why companies must always follow at least the recommended best practices. If they are followed, the target might be too hard to break into and a hacker might go someplace else where it's easier to break in. Targeted attackers might still get it, but we should all make sure they have to work DAMN hard to succeed! If we start thinking that the attackars will succeed anyway, we might as well drop all defences. Display the admin passwords at the bottom of the "About us"-pages.
Bottom line, HBgary fucked up good. They showed the world that they give advice that they don't follow. They deserve the burn and anybody thinking about hiring them should think again. Even if they change the problems that allowed this breach, the basic problem is that they obviously don't understand security. If they did, none of this would have happened.
</end rant>
I can understand a typical organization making most of these mistakes, but a security firm?
This might be a "cobbler's kids shoes" issue, or just a general failure of people and process.
One of the only truisms I've found so far when dealing with breaches is that almost no one gets this right proactively. You almost have to be the victim of a breach (the more public the better) to actually rethink how your people/processes are implemented.
This seems true for the largest banks in the world, and the smallest security firms.
And it's easy to look back in hindsight and say "how could they have possibly had things set up that way?", but the truth is that this was not an opportunistic attack; if these vulnerabilities weren't present, then they'd have looked for others.
That's the problem with securing your environment, you have to get everything right, and the attacker only has to get one thing right. That being said, as with most "disasters", this was a series of cascading failures (like most airplane crashes, or oil rig explosions).
Hopefully they'll learn from this (assuming the negative fallout doesn't completely bankrupt the company).
If it's ever possible for me to hire a security firm that has higher standards than this, I'm going to do that!
When people at a company don't do this, it's often a symptom. A friend of my girlfriend worked at an AT&T store. She could've gotten a huge discount on AT&T mobile? Her answer: no thanks.
* An initial entry point through SQL injection (from which we can infer that the federal site was probably never security tested)
* The use and re-use of weak administrative passwords
* The adoption of poor practices (such as sending passwords in cleartext via email) at rootkit.com
Of all of these, the attack would've probably been limited to the federal site had Aaron used decent strength administrative passwords (or just not reused them). The issue regarding the website comes down to risk ownership. If Aaron Barr was the risk owner for the website, it falls down to him. If in his contracting the company, Aaron had a plan to test the website then whomever tested it missed the SQL injection (which could've been implemented after a test by the CMS firm without him knowing, or any number of things). If Aaron didn't have a plan to get the site checked out, then that's both strikes falling on his shoulders. The third strike is quite clearly stirring the hornets nest.
Moral of the story: Don't antagonise groups that will strike back without a plan to deal with it.
Moral #2: Don't reuse passwords.
Moral #3: Retest your app on every significant change (including initial deployment).
I'm sorry but thats just egregious. Thats like being a bodyguard and not even putting a lock on your own house.
The rest of the attacks could have happened to anyone. We all know its best practice to use many different passwords but most don't because its more convenient to only have one or a few. And if the email is coming from the email address it should you could brain fart and give up the info without thinking.
But the first two parts of the attack should NOT have been possible for them to even pretend to call themselves a computer "security" firm in 2011
This is what we do. We use Google Apps so we used a combination of existing policy, crypto and user awareness. It's not the use of email that's an issue, it's the how the data is stored. If it's encrypted with good crypto it's not a problem. If it's encrypted with bad crypto or no crypto then the extent of the problem is down to the data.
As an aside, while you would want to encrypt anything sensitive, that doesn't mean you need to encrypt everything - it certainly makes conversations over smartphones more difficult, and google chats wouldn't be encrypted.
Still, a little common sense goes a long way.
Also, a common policy of encrypting and signing emails would have stopped the social engineering attack completely, as the sysadmin would've known not to accept an unsigned request to give out passwords.
Kind of mind boggling that people don't do this generally already.
There must be more to it than this. If you know it's one of two passwords, why bother asking - couldn't you just try both? (In retrospect, maybe it was to give Jussi confidence that he was communicating with the real Greg? [Who else, after all, would know the root passwords?])
I once published all my root passwords on IRC as a challenge and didn't change them for a few weeks.
Nothing happen.
HBGary used Google Apps for its e-mail services
The more I learn about this incident the more Mr. Barr and HBGary look like a bunch of amateurish dolts that may give good power point presentations, but I for one sure wouldn't take my security business there.
We moved to Google Apps as they bought Postini. We had an adult discussion of the benefits and drawbacks, and as everyone had PGP made it a straightforward job of encrypting things according to policy. Google Apps (for business, at least) offers SSL encryption on everything and offers relatively little by way of additional risk compared to using someone like messagelabs for your AV. Obviously they're storing the data for you, but you'd get that with any hosted provider.