Hetzner Servers Compromised | Hacker News Reader

Hetzner Servers Compromised

wiki.hetzner.de

180 pointsmartindale130 comments

As part of the registration process with hetzner.de, you have to send them scans of personal documents (such as passport, drivers license or similar).

I asked them just now if these systems were compromised and they promptly replied:

"The system that stores scans of ids, credit cards and so on was not compromised. In addition to that, we delete that information after 21 days."

"you have to send them scans of personal documents (such as passport, drivers license or similar)."

What? Seriously?

"In addition to that, we delete that information after 21 days."

Hmm, do they delete the info that ends up on backup copies? How do you know they even actually delete it in 21 days? It's not like there is a third party even auditing which you can rely on. (Not that I'd ever do that for something like this anyway, I would just find another provider.)

Yes seriously, here is what they asked me on first order with them:

"Since you're a new customer with Hetzner, we ask you for a scan of your passport or ID card (authenticity check). It's only necessary for your first order with us.

Please send the scan by fax or as an email attachment."

When they say they delete it after 21 days, as they did in the mail I've just received, I trust them. I find their communication on this matter, as well as previous matters, open and serious.

not true for all customers - I needed to send them nothing of the kind (in the US)

This was dedicated servers (root servers they call them) - and I'm from Europe.

they haven't been doing this forever, so if you've been a customer for a while, you might not have been asked.

they also don't ask business customers (might not be true for all countries) if they supply certain details about their business.

Hey - I didn't have to provide anything but a CC number and I'm based in Europe. We don't have a huge account, how many boxes did you order?

I did need to send them scans of ID documents and I am in the US.

If they're proper backups then they're offline and you don't have to worry about them getting hacked.

They might be stolen.

Okay, but stealing a backup gets you 21 days of those documents, the same amount you can get from the live system. There is no need to worry about backups in particular.

Correct me if I'm wrong, but if you had a backup from 15 days ago, wouldn't that backup contain documents 21 days prior to it? So in effect: up to 36

The backup will have documents from 36 days ago through 15 days ago: 21 days.

If you're worried about someone stealing your entire backup system then you have bigger issues.

Is this outside of Germany? I rent two servers and I never sent any ID.

This shit is annoying. I think it have been only 6-8 months since the managed server part of Hetzner (KonsoleH) got hacked. Now the VPS/root server part (Robot) got hacked. I understand that both incidents are completely different and it seems that they might've learned a thing or two from the KonsoleH-hack, but still. My address data and my bank data are very likely to be compromised.

But then, changing the hoster doesn't make any sense. My data is somewhere out there, can't get any worse I guess.

One of the differences between servers in data centers and desktops is that servers don't reboot a lot. So the bad guys can build an in memory system that doesn't change anything on disk (which avoids the configuration management system from flagging it) and if it cloaks itself as a normally long lived process then active monitors might be fooled as well. It then runs, effectively undetected, until the server reboots which can be months (or even years).

Tools to protect against those threats are going to have start taking into account process activity and footprint.

Hetzner is very open, transparent, and exact in their communication about this. I'm not a customer but I really like that they didn't try to dress anything up and were forthcoming with information.

Doesn't surprise me. I'm a customer and the whole experience feels like this.

Several events with Linode, now Hetzner. These are relatively "premier," high-quality hosting companies, you can count on thousands and thousands of companies to pay even less attention.

Yet every time, the discussion is only about one specific company, without seeing any broader pattern.

When are we ever going to draw the conclusion that popular hosting companies (and, actually related, facilities like RubyGems) are especially attractive targets, and that the approach of waiting for an exploit and then shaming the targeted company is not an effective way of getting better security?

Apply Occam's Razor. The simplest speculation is an internal breach. All the procedure in the world won't save you from that.

They are of good quality, but let's keep in mind that Hetzner tends to be solidly on the budget side. I am a Hetzner customer, and it stinks that this happened, but let's consider: I don't expect Bugatti quality when I'm paying for a Chevy Malibu.

I do appreciate them being forthcoming and sharing some security details that do make me feel pretty safe. I don't care if they have the last few digits of my CC number, and I've already reset my password. Stinks I had to even think about this, but again, Chevy Malibu.

Assuming that the exploit really is unknown, what better security could they have done to prevent it, without severely impacting usability?

Compromises will happen sooner or later when running a large number of public facing services. With good policy, the breaches will be well contained, which seems to be the case here. What more can be expected out of a premier hosting company, let alone Hetzner which is very cheap budget provider?

Is it just the customer account details that apparently make hosting companies attractive targets. If that's the case, I'm wondering why we're not seeing more breaches from all over the e-commerce world. Why just hosting companies?

Maybe the hosting companies are simply more likely to detect a breach. Not a happy thought I know.

And more likely to disclose it perhaps?

There are a lot of breaches that you will never hear about because they go undetected or undisclosed.

The info in the mail regarding the safety of credit card info contradicts with the linked FAQ.

FAQ: Bank details are encrypted (two-way) in the database. However, it cannot be excluded that the attacker/s have also been able to obtain access to the key.

Mail: With credit cards, only the last three digits of the card number, the card type and the expiry date are saved in our systems. All other card data is saved solely by our payment service provider and referenced via a pseudo card number. Therefore, as far as we are aware, credit card data has not been compromised.

banking details is a much broader term than credit card number.

account numbers / routing numbers / company names / po boxes/ physical addresses / account history and other billing info.

I'd assume they're saying some of these might be compromised, but specifically that credit cards aren't.

Hetzner probably has many customers who pay with debit card/recurring direct debit instead of credit card. It's quite common in Germany.

They also allow PayPal, which I recently changed to with Hetzner. 99% of my payments to websites are PayPal, for this very reason.

Perhaps the email is different based on that particular customer's payment method, which is why one is talking about credit cards and the other about banking details?

Sigh, another hosting provider hack. As if Linode and OVH are not enough. This is the reason why we use full disk encryption, where we enter the key manually during boot. This way we're protected against many types of hosting provider hacks.

What kind of effect does this have on your I/O performance?

Minor. It costs more CPU because of encryption/decryption. But our servers have so much CPU power, and our workload is mostly dependent on disk I/O throughput, so enabling encryption is almost free.

AES-NI features in modern CPUs reduce it even further.

What types of hacks does that protect you from exactly, considering you don't have physical access and can't even see the hardware?

It protects against hackers that compromise the provider's administration interface, and then use that interface to reboot the server to single user mode or a rescue system. Upon reboot, the attacker will encounter an unusable blob as filesystem because he doesn't have the encryption key.

It also protects against hosting provider employees that detach the hard disk and attach it onto another machine to copy data off it.

If the server is virtualized, then in theory it's still possible that the attacker hacks into the hypervisor, and then through modifications in the hypervisor's RAM gains access to the guest kernel, and thus the guest system. However the skill that is required for this is an a whole new level compared to "regular" hacks.

If the attacker has physical access then it's still possible that he uses electromagnetic emissions from the hardware or power supply patterns to obtain information about it to hack the system. But again, this requires skills on a whole new level.

Very good point about single user/rescue, thanks.

Do you have a very small number of servers? Requiring manual steps to boot is not very scalable--if there is an event that requires all your servers to reboot (power outage, mandatory upgrade, etc) you would be in quite a tough spot.

I much prefer having to repeatedly enter the password dozens of times than risking data leaking out - or worse, data being tampered with.

So "yes" – what would you do if you had, say, several hundred (or several thousand) servers?

If I have that many servers then I can assume I have a large budget for security, right?

I would make a password entering automation system. Ensure that that system is dead-simple and secured to death. It must run no other services, firewalled to death even from the intranet, physically secured in a cage, and must be off most of the time. It is only to be turned on when booting a system, and turning it on not only requires a password but also physically walking to the cage, opening it with a physical key, and pressing the "on" button. All target servers must be configured in such a way that they can obtain network access before mounting the encrypted part.

But I'm not a security expert. Maybe I've overlooked something.

Wait, this sounds way harder than the previous solution and entering the password over a remote IPMI console.

It's harder to set up. But when the password entering automation system exists, you just grab the key, go to that system, unlock & boot it, then tell it to enter the passwords for your 1000 servers.

With that many servers you probably have to reboot a few of them pretty often.. You'd have to live in the datacentre :)

I'm not sure you can assume that if you're, say, Instagram (pre-big-round/acquisition). You may well have several hundred, even thousands, of AWS instances with a staff in single digits.

Properly hard problem, as you've pointed out.

Shouldn't all your servers have different passwords? How do you manage the passwords in case you're not available when the server reboots?

Have two people know all the passwords. Let each of them encrypt this list of passwords with a password only they individually know. Sync passwords each time someone changes something.

How does this help? The key is still stored in memory which I assume the hypervisor has access to.

Correct, but hacking into the hypervisor is harder than hacking an administration interface. In the end the only secure server is one in a vault at the bottom of the ocean, but there are ways to prevent certain attack vectors.

>In the end the only secure server is one in a vault at the bottom of the ocean[...]

I strongly doubt that:

https://en.wikipedia.org/wiki/Glomar_Explorer

Hetzner provide a lot of physical machines too, I believe this is what the other poster was talking about.

Physical machines don't prevent keys from leaking out. A physical attacker can analyze power usage usage patterns to extract the encryption key. :)

I don't think you have that sidechannel with AES-NI. Besides, as a physical attacker cold boot attack would be much easier Or if the server has any interfaces with DMA, like PCI or something, that's even easier.

Wouldn't that require physical access?

Ah, I missed the physical machines. No hypervisor to crack, just out of band management cards. :)

What if attacker infects unencrypted part used for booting? Do you have protection against that?

It's relatively easy on Linux, where /boot is usually not encrypted.

Agreed, but typically /boot isn't automounted when it's a separate partition, which means they would need a root-access compromise already.

They can replace the booting kernel. Unless you use SecureBoot, TPM or some such thing, there is no way to protect against that (assuming that the attacker has access to the shut-down system at one point and you boot it later).

Hetzner allows to netboot any server via admin panel. After booting into recovery os, you can modify unencrypted parts as you wish.

The trick is putting all important data in the encrypted part. The unencrypted part must be reconstructable in an automated manner. If you are paranoid and you assume that all unplanned reboots are attacks, then you can reconstruct the unencrypted part every time. Tools like Chef make this relatively easy.

The kernel lives on /boot, and could be subverted to lie about its uptime to convince you that there was no unplanned reboot.

This is the same methodology I use, and I think about these attacks a lot. There is no good/cheap way to verify a remote execution environment right now with commodity hardware. :/

That's not possible. Even if you fool the kernel, you still can't mount the encrypted disk. The fact that the encrypted disk is not mounted is proof that there has been an unplanned reboot.

Wow, do you actually reconstruct /boot on every unplanned reboot, yourself?

I don't know. Ever since we set up encrypted disks we haven't had an unplanned reboot. I was simply pointing out that this is a possibility that we've considered.

Well, maybe. Do you have servers with a secure boot environment?

Also, if the attackers are able to attack your servers in a similar fashion (gaining access without reboot) -- they have full access to the already decrypted disks (well virtual devices).

As for other comments on automating booting -- as eg cryptsetup allows entering the encryption key via ssh -- automating that shouldn't be too hard.

I still think that it mostly defends against someone stealing/cloning the physical disks though.

Even without secure boot environment, if you're paranoid enough you can throw away and reconstruct the boot environment on every unplanned boot.

Encrypted disks are not meant to protect against attackers that can gain access to the server without rebooting. But all your OS-level security mechanisms (user separation, AppArmor, file permissions, firewalls) become useless if the hosting provider's admin interface has an exploit and the attacker can reboot the machine into single user mode. In other words, the Linode scenario. That's what encrypted disks protect against.

My point was that getting back to a known good boot scenario isn't all that easy if you don't have physical access?

I suppose the sanest middle ground is to wait for the dust of an event like this to settle (or have someone investigate, in case of "just" an unexpected reboot) -- then wipe the server(s) in question, reinstall (hoping the install media is ok) -- and importing the disks/backup.

I still think it is little gain for a lot of work.

Basically if your server software is secure, you've avoided data compromise due to a reboot into single user mode or similar -- but you have to assume (for the sake of guaranteeing the safety of the encrypted data) -- that attackers have a copy of the data at rest (which they can get if the can boot into a recovery image etc.

So now you have to make sure that you can boot a safe environment in which to import the data to, without leaking your key(s).

All the while, it is rather likely there are a few vulnerabilities in your software stack that all this never did mitigate.

I'm not saying don't do it -- I'm in favour of doing it for the (relative) peace of mind that no that will get lost/out as the servers are recycled -- I'm not sure it makes much sense to avoid hackers -- the online threat seems to dominate for most cases I can think of.

Full text of the email sent to cutomers:

Dear Client

At the end of last week, Hetzner technicians discovered a "backdoor" in one of our internal monitoring systems (Nagios).

An investigation was launched immediately and showed that the administration interface for dedicated root servers (Robot) had also been affected. Current findings would suggest that fragments of our client database had been copied externally.

As a result, we currently have to consider the client data stored in our Robot as compromised.

To our knowledge, the malicious program that we have discovered is as yet unknown and has never appeared before.

The malicious code used in the "backdoor" exclusively infects the RAM. First analysis suggests that the malicious code directly infiltrates running Apache and sshd processes. Here, the infection neither modifies the binaries of the service which has been compromised, nor does it restart the service which has been affected.

The standard techniques used for analysis such as the examination of checksum or tools such as "rkhunter" are therefore not able to track down the malicious code.

We have commissioned an external security company with a detailed analysis of the incident to support our in-house administrators. At this stage, analysis of the incident has not yet been completed.

The access passwords for your Robot client account are stored in our database as Hash (SHA256) with salt. As a precaution, we recommend that you change your client passwords in the Robot.

With credit cards, only the last three digits of the card number, the card type and the expiry date are saved in our systems. All other card data is saved solely by our payment service provider and referenced via a pseudo card number. Therefore, as far as we are aware, credit card data has not been compromised.

Hetzner technicians are permanently working on localising and preventing possible security vulnerabilities as well as ensuring that our systems and infrastructure are kept as safe as possible. Data security is a very high priority for us. To expedite clarification further, we have reported this incident to the data security authority concerned.

Furthermore, we are in contact with the Federal Criminal Police Office (BKA) in regard to this incident.

Naturally, we shall inform you of new developments immediately.

We very much regret this incident and thank you for your understanding and trust in us.

A special FAQs page has been set up at http://wiki.hetzner.de/index.php/Security_Issue/en to assist you with further enquiries.

Kind regards

Martin Hetzner

Have all Hetzner customers received this mail? I currently have a couple of servers with them and have received nothing yet.

AFAIK yes - it might take a while though, i've gotten mine quite a while after the first ones seemed to have gotten them..

I also got the mail.

I just got mine now (almost 45 minutes after this submission to HN) so yours is probably still in the queue.

I did not get it yet. Perhaps they are only sending to those who might be affected?

yes, received mine this morning as well.

Thanks all, finally got my mail. I was worried my account was hijacked! Panic over

I can also confirm receiving the email (but I really need to tidy up my mail filters, I actually read about the hack here first).

Nice email, one improvement would be to host the wiki behind a proper https-site -- especially since we already know the attack appears somewhat sophisticated, and that the attackers gained control over servers on the Hetzner network. Now, the faq doesn't contain anything obviously bad (email your credit card info to... etc) -- but one small thing that could be improved.

Other than that I appreciate how Hetzner have handled this so far.

Just got the email too. I'm quite happy with how clear they are about what happened and how they hash their passwords. Very different from Linode.

I just wish that they used bcrypt instead of a salted SHA256 but at least it's salted (and anyway in my case I never reuse passwords so no big deal).

> I just wish that they used bcrypt instead of a salted SHA256 but at least it's salted (and anyway in my case I never reuse passwords so no big deal).

If they fall under PCI:DSS, they might not be able to use bcrypt since it isn't an official recommended standard. (I am of course assuming that they mean PBKDF2 when they say salted SHA256.)

Would they be able to get around this by encrypting it two times? On of the times with the official recommend standard an another time with bcrypt?

https://www.pcisecuritystandards.org/pdfs/pci_dss_glossary_v... seems to list Blowfish as an approved standard as far as the PCI standards group is concerned.

Thanks for that link. I think that means blowfish is approved as a standard encryption algorithm, but that doesn't mean it is approved as a standard way to store passwords.

From http://security.stackexchange.com/questions/4781/do-any-secu..., I gather this: bcrypt is neither a NIST or FIPs standard, but PBKDF2 is.

> For any business required to comply with U.S. NIST or FIPS standards, bcrypt is not a valid option. Check every nation's laws and regulations separately if you do business there, of course.

However, from a comment on this other answer: http://security.stackexchange.com/questions/11552/would-it-m..., someone mentions this:

> As far as I know, PCI doesn't specify actual technology, especially Encryption specifications. The legal cases I've seen support what's technically feasible, not technically ideal or perfect. I'm sure either one [bcrypt or PBKDF2] meets PCI requirements

An opinion shared by many people that deal with PCI, is that you follow PCI not to secure your systems, but to limit your exposure to PCI fines and remedies. It seems like it is a rock-solid defense for PCI compliance to choose the algorithm that is certified by NIST.

But it is also likewise plausible to say that you're using a newer technology that's understood by experts in the industry to be even better than those certified some time ago.

I agree with you, but there is zero risk w/r/t PCI:DSS if you use the FIPS standard.

The difference is that one way your lawyers won't even need to make an argument and the other way they might have to make an argument.

That's quite an assumption.

Could you double hash passwords? First with bcrypt for actual security, and then a second time with SHA256 as a bureaucratic measure?

I have a link above about about using PBDKF2 and bcrypt together http://security.stackexchange.com/questions/11552/would-it-m...

The user Thomas Pornin knows his crypt which is why he has 86k reputation.

Would it matter? I mean all they have to do is encrypt it using the old standard and then encrypt that using bcrypt.

Lawyers "do you use blabla standard to encrypt the data?" You> "yes" - and it is secure.

I don't understand, are they saying the backdoor might be because of the Nagios software or because something has implanted itself into the Nagios software?

I am not a Hetzner customer, but the comment about Nagios perked up my ears. I work with some clients who have Nagios running in their environment, and I'm wondering if there is an exploit in Nagios or if it's just a coincidence that this was where they noticed the infection?

It makes me wonder if there's any connection with the recent Drupal Security problem (they cited a "third-party software installed on the Drupal.org server infrastructure" but they haven't - afaik - disclosed the software name yet)

Nope, there is not, the two are fundamentally different.

Can you clarify why you believe this, or what special knowledge you have?

It's most likely that Nagios was simply used as a tool. It's easy enough to add a Nagios plugin that does something naughty and call it check_inode_usage or something.

..not that I'm a fan of Nagios' architecture though. I'm not.

I suspect it's simply out of date. Nagios does have a large surface area, but from what I've seen in the past, monitoring systems are very difficult for sysadmins to want to upgrade. :)

I'm kinda confused about why Nagios was even involved here. If you've got all your checks setup properly, the Nagios host doesn't have any write access to your monitored hosts. You can use NRPE to restrict what commands it runs. Combine that with a read only SNMP account, and even if your Nagios server is compromised it cannot access your server.

You might be surprised at the number of incorrectly configured Nagios installations.

Well it's hard to tell what Hetzner means, it could be someon just piggybacked onto Nagios, but there there is also this, http://www.cvedetails.com/vulnerability-list/vendor_id-1424/...

Huh, this doesn't make me happy. I haven't gotten an email yet.

"The standard techniques used for analysis such as the examination of checksum or tools such as "rkhunter" are therefore not able to track down the malicious code."

rkhunter is out of date.

http://rkhunter.sourceforge.net/

http://en.wikipedia.org/wiki/Rkhunter

We use it as well but I'm not sure it is an example of a serious security practice when given with only one other "checksum or tools such as" type statement.

Considering https://twitter.com/omgtbh/status/337567604887658496 I can't say I'm surprised...

(tweet text reproduced here: "I asked Hetzner if they plan to support 2 factor auth & was told that they already do - they require a username and a password. Seriously.")

You don't seriously think that a server admin said that, do you? It was probably some low rank customer service peon.

Of course not, but regardless of who in the company said it, the official support response was as quoted. Not exactly encouraging!

To be fair, Hetzner has very "google translaty" English support for simple matters, so it is not impossible that they simply didn't understand the question

Exactly, and somebody who didn't yet hear the term can easily come to the conclusion that it is an authentification with two "factors" which means basically two things and that would be username and password. In the past there were system with only one factor...

Kinda reminds me of http://www.youtube.com/watch?v=gmOTpIVxji8

Things seem to be getting worse there. A server we have there just went offline, and their management servers are not responding. I wonder it's part of the exploit or as a result of attempts to "fix" the problem

I had 5 minutes of downtime, but everything seems to be back in working order.

Am I an overly-cautious nubcake, or do any of you also refrain from clicking the link when it says, "We've been compromised. Come visit our site!"?

If I were a mean dude who hacked into some prominent site, the first thing I would do is submit a link to HN that points to a malicious "article" and laugh as malware gets thrust into thousands of computers.

Way too much mork. If you have a way of infecting computers by simply visiting a webpage, just tag it some kind of porn and post on reddit & other forums.

Seriously, again??

I think such a fauxpas shouldn't be tolerated twice.

Do you have link to their previous hack?

In October 2011, unfortunately these links are in German only:

http://www.heise.de/security/meldung/Web-Hoster-Hetzner-geha...

http://www.heise.de/ix/meldung/Datenleck-bei-Hetzner-1356468...

http://www.heise.de/security/meldung/Passwortklau-bei-Hetzne...

There was unauthorized access to customer data, even passwords. Someone claims to have accessed it via an FTP-server where he found a root password to a management server

Thanks for posting the links, I've forgot to mention it in all my wisdom, duh..

Here is the (english) mail they sent to customers back in October 2011 on the previous intrusion:

http://pastebin.com/uJhG5aLS

The lesson to take from this is you can't rely on someone else providing security for you. You need to make sure you "layer" your security. If your database is compromised make sure you are salting your passwords or at least using a hash that is designed to be resistant against cracking.

This doesn't protect against someone who has root access to your box and manages to capture passwords in the clear as they are being transferred but it does limit the surface of a breach.

Something was happening with their routers, too. Using PingPlotter to one of my servers shows severe packet loss on the hos-bb2.juniper2.rz19.hetzner.de router.

I see it just cleared up for now. Still, I'm moving my traffic to other locations till this blows over.

That might be because various parts of germany are flooded at the moment. One of Hetzners datacenters is located in Falkenstein near Regensburg and others might me affected too

I've got a login for robot.your-server.de and have changed my password but don't seem to have one for konsoleh.your-server.de.

Is the konsoleh login/account something that would have gotten created for me automatically? i.e. do I need to worry about it?

You need a konsoleH account for the registration of 'special' domains not covered by the robot. One doesn't have one by default.

I think webspace users get a konsoleH account, too.

KonsoleH = managed servers

Robot = VPS/root servers

I use the Robot to access a VNC console of my leased server. I can even do a CTRL-ALT-DELETE and boot the OS in single user mode from it, if I only have access to the Robot.

Doesn't this imply that my server is potentially compromised as well?

hmmm... what does Hetzner do?

They're a German Server/Webspace ISP - see www.hetzner.de/en

Not just German actually, they (Hetzner Online AG) have also got a subsidiary in South Africa ... http://www.hetzner.co.za/

ik,r

But since this seems to be affecting only the main/german division, i didn't bother to mention them.

They are a provider of hosted dedicated servers, colo and virtual machines (with or without server management services), based in Germany.

They're really inexpensive, which explains their niche popularity...

They're very big, not just a niche. Also reliability / support is not necessarily always so much worse than some higher price solutions.

Aye, they seem to have a very good reputation for a cheaper provider. I've not used their services myself though.

Their SSL certificate is fucked for their security page... oh dear.

https://wiki.hetzner.de/index.php/Security_Issue/en

Hope everyone has fun with their identity theft cleanups. They ask people for ID scans.

Bank details are compromised.

boo!

This was announced AFTER the German work day was done. Really nice for admins. Thanks.

You would have preferred that they waited until tomorrow?

Reply on news.ycombinator.com