Dreamhost asking users to change passwords after database breach | Hacker News Reader

Dreamhost asking users to change passwords after database breach

dreamhoststatus.com

124 pointshopeless48 comments

I'm a Dreamhost customer and haven't received any emails regarding this yet.

Dittos. I found out on HN. Checked to see if it had gone to some stray account / folder, but I never received any notification that I can find. Anyone?

nothing. I've discovered I had the same pwd for the panel and for the ssh access, although I almost never use the ssh one. Time to check every password, it seems ...

It would also be very, very, very cool if Dreamhost (since they have a list of old passwords) would, say, make that list all invalid for future use.

We're doing some tough client love right now on security practices, and having a nice hard wall to push off of, namely, Dreamhost being assholes, in a good way, would be a very Good Thing[tm].

We, well, I, have been using sshkeys for access. We actually had one of our accounts request sshkey access as well, which was freaking wonderful. Most are so technically backward that this can't be rolled out globally, though I'd love for it to be.

We're not quite done investigating the issue and resetting passwords; once that's done, though, we will have a mass email going out.

Well, I was using the same password for gmail and dreamhost, because I figured both were secure. Yeah, I know that's bad practice, and I don't do it anymore (I use a password manager for new sites), but I'd set up my dreamhost account a while back, and forgotten I was using my "secure" password.

I won't be the only person doing this.

I'm confused by the directions. If I have 20 usernames did you reset all 20 passwords for these names to random strings and now I just need to pick new passwords of my liking on my own time?

Or do I need to go through all 20 right this moment and change them from their old value to a new value?

Basically do the hackers possibly have access to my ftp accounts or have you already switched my passwords to random strings?

A mass email going out this morning so I could have got on the ball with this and coordinated response with our client management folks would have been a Really Good Thing[tm].

As it was I found out about 4 hours after your first blog post via HN.

We're still hashing out what we're going to do with folks who, last time we instituted a password/process change, wanted a 3-weeks heads-up.

FML.

FYI: The forgot password feature does not seem to be working at this time.

If you log in with your Web Panel password, you can change an accounts' password without needing to know the old one.

The DreamHost engineer who's been commenting here says the web panel passwords haven't been compromised (I changed ours anyhow).

Same. Almost 24 hours ago and this is the first I've heard about the breach.

Also, the fact that you can see your user passwords in the panel has always irked me, and that has gone unchanged even after past breaches. I have been quite happy with their service, but maybe this my queue to leave... :(

EDIT: To their credit, I see this when logging in to the panel: "Due to some unauthorized activity we detected within one of our databases, we have forced a reset of all FTP/SFTP and shell passwords as a precaution."

We've got a mass email going out now.

Neither. And ironically I was just logging in to grab everything I had there to move it to a new server, and couldn't log in, emailed them with no response in the last 2 hours then see this here..

This is pretty bad. I've received password reset emails from dreamhost in the past and the passwords are in plain text...I just renewed two days ago too.

Hostgator does this too and don't reset, only resend passwords. It has always bugged me. There is no reason to be storing plain-text, especially for their billing system.

At least Dreamhost says the Shell passwords are hashed, which makes sense.

I didn't know that about the plain text dreamhost web-panel passwords though.

Surely they don't send you your old password, but a freshly generated one? Then they could still be hashing them after emailing you.

Embarrassingly... no. Our login/authentication system was written in 1999, and it shows -- we store panel login passwords using symmetric encryption, and send out the decrypted password when you request it.

Getting this fixed was already on our to-do list. This incident has moved it up to near the top of the list (competing with a few other security-related tasks).

I have always been bothered about cpanel passwords coming through in plain text. To confirm, is this the same storage system with mail passwords also?

Shell passwords - they're hashed, but are they salted? If not, can they be in future?

Thanks for your time.

I've been happy with Dreamhost's service, but becoming aware of this in the last few months has forced me to look into other registrars. If this is fixed, I would be much more inclided to stay.

If you could forward these articles to whoever's working on security, I'd appreciate it (and they're a good read): http://www.codinghorror.com/blog/2007/09/rainbow-hash-cracki... http://chargen.matasano.com/chargen/2007/9/7/enough-with-the...

I have limited web development experience so maybe someone can fill me in. Why is securing a web server so difficult? I understand they are complicated systems with many technologies, but large companies with lots of smart engineers can't seem to solve this problem. Can most of these hacks be attributed to user error?

It's a combination of things, but mostly, every piece of software expands your attack surface. Most systems are not written with defence-in-depth and most have all-or-nothing authentication/authorisation.

For shared hosts it's worse: a flaw that punches through can affect many customers at once. And there's often a larger attack surface because of the complexity of hosting multiple customers in a single system.

Take, for example, Wordpress. It uses a single login into a relevant MySQL database. Plugin code has the same level of access to MySQL as the core Wordpress code. If a there's a defect in either, your MySQL database is wide open to receiving SQL statements.

If you didn't configure the database properly, that wide open pathway now leads to other databases, or to shell access with interesting privileges, and so on.

So the design of wordpress (and in fairness, pretty much all web apps follow a similar pattern) both increases the attack surface and decreases the depth of defence.

This is based on my own biased opinion...

It can happen for a number of reasons and isn't necessarily with securing a webserver.

Sometimes security issues get that "well, it isn't an issue right now" view and get put in a backlog and never resolved. Other cases it is a lack of good code reviews or security reviews. Or inexperienced developers (at least inexperienced with that kind of security).

From a Dreamhost employee (http://news.ycombinator.com/item?id=3491643), that code was from 1999. Could have been any one of the above.

The truly sad part is the number of examples of this kind of thing blowing up yet companies are still making the same mistake. At what point does someone think of checking their own apps password practices to ensure they aren't next?

One practice that's starting to help, a bit, and it pains me to say it, is security audits. There are a few standards emerging for SAAS services, especially at the B2B level. And as startups (or non-startups) start intersecting with large, regulated, publicly-held businesses, these requirements propagate, and with audits, receive multiple reviews.

The process is far from perfect, and many of the standards are laughably lax (and yet ... they still aren't met). Because they're standards, the requests are fairly uniform. A number of issues regarding Dreamhost have been on my own back burner for months, and I'm finally getting that Round Tuit this afternoon.

Pushing for good, solid standards would be a net benefit. That's the silver lining here.

This sounds promising. I've seen the answers on stackexchange and quora about web security, but until my site is tested by a hacker it is difficult to know if I've done everything properly. So I like the idea of some kind of audit to help in this process. Maybe hiring some white hats to pound on my site would be a good idea.

Mentioning standards and security audits makes me cringe in their current state. Certainly agree with you that they'll help, but many are draconian. Things like physical access lists when you use cloud services... or policies on tape backups. :)

There's a lot of dumbth in them, to be sure.

I'm hoping that they'll evolve toward slightly more sanity than insanity.

Some institutional laws suggest that this is wildly optimistic on my part. Security theater exists because bureaucrats and legislators want to appear to be doing something. Vendor-based solutions are mandated because vendors have political clout. Effective tools are difficult to deploy in real-world scenarios -- filled with failing devices, intermittent communications, poor training, and worse end-user understanding.

But give me my dream. Just for this one night.

Securing them properly is pretty straightforward, but it takes time and skill. Time+Skill = money. Web hosting companies compete primarily on price because its the only thing that is easily compared. How does one measure a service provider's competence? Like used cars, the quality ones get pushed out of the market by the dirt cheap lemons.

It's more cat and mouse, as complexity grows with computing power, there is a gap between the state of the art, and commodity systems that are sold for profit in global markets. Because UNIX works so close to the hardware, it is especially prone to novel methods of attack if left unattended.

From the comments, it seems the passwords were hashed but they're (rightly) still concerned.

Unfortunately, there still seems to be some confusion about when/if the password reset has taken place, and whether users should change them now, wait for Dreamhost, or both (and then update them again after the reset!)

We're currently in the middle of resetting passwords, machine by machine. If your old password has stopped working, that means we've gotten to the machine that you're on, and you're safe to reset it through the panel.

Panel passwords are safe, BTW. The only passwords that we believe may have been compromised are UNIX user passwords (e.g, FTP/SFTP/SSH users).

Unix passwords are Never safe and never will be.

I don't get you guys...

I just got the email from Dreamhost at about 3:30AM EST. I checked out their status blog and found something interesting: over the past month a good number of people have had malicious .htaccess files and scripts inserted into their Wordpress installs. That would include me.

About a month ago one of my Dreamhost sites was hacked. An .htaccess file was modified to insert some PHP scripts that provided a backdoor to the site and that also embedded ads. I see now that I'm not the only one who has had this problem. When I reported this at the time I submitted as much information as possible but only got a canned, autogenerated response. Dreamhost ran an automatic scan on my site and IIRC it did pick up the offending file, but by the time that had been done I had already found it. The particular backdoor was c99shell.I. As I've read more comments, people have had .htaccess files modified who weren't using Wordpress, so it's not a Wordpress problem.

This leads me to ask: Is this compromise related to the hacks that I and other users have experienced? If so, how long has the password database been compromised?

Thankfully I use GMail for email, so at least that level of personal information wasn't stored on their servers. But this makes me wonder just how long someone has had free reign to access many Dreamhost accounts.

Needless to say, I will be switching hosts and will never recommend Dreamhost again. I guess you get what you pay for, and if I had been able to speak with an actual human being about my site being hacked maybe they could have connected the dots. Furthermore, they've had this weak password system since 1999?

Edit: Here's a link to the status blog post: http://www.dreamhoststatus.com/2012/01/20/changing-ftpshell-...

Edit2: I just remembered that I got a strange call from a friend a few months ago who said that when he tried to access my site his corporate firewall had blocked it because of the presence of pornography. This is a church's web site we're talking about, so of course there was no porn on it. This is the same site that was later more overtly hacked in December. Apparently the initial hack only inserted text when a crawler hit the site, so it was an SEO hack. In December, the text became visible to regular users of the site. This post has more detail: http://www.dreamhoststatus.com/2012/01/20/changing-ftpshell-...

After a few emails with DH support, I discovered that there was a file created by another user in two of my accounts. Somehow the permissions for two directories had been set to 777, and a compromised user put a trojan in them. I think that I jumped into conspiracy theory territory above, and misrepresented Dreamhost's security and response as a result of that. The combination of my emotions and my inexperience with dealing with hacking on web hosts led to an incorrect judgment.

Do only shell and ftp passwords need to be reset or does the main account web panel password need to be reset as well?

Ask because, we don't use shell/ftp passwords and I'm not sure if we are affected or not.

Just shell/FTP passwords. As noted above, panel passwords are fine.

I am absolutely not shocked at all. I got off their service years ago when they had all their FTP accounts breached.

Hostgator is no better. Talked with "technical support" couple weeks ago; decided to save the chat:

Welcome to GatorChat! You are being connected to a representative in our Technical Support department right now.

(01:33:17 PM) Joe: hi there.

(01:33:26 PM) Nathan: Welcome to Hostgator LiveChat. My name is Nathan. How may I assist you today?

[...]

(01:38:30 PM) Joe: never got password

(01:39:20 PM) Nathan: Its the same as your billing password.

(01:39:46 PM) Joe: YEAH

(01:39:48 PM) Joe: I JUST GOT IT. WHICH MEANS YOU GUYS KEEP PASSWORDS IN PLAIN TEXT

(01:41:01 PM) Nathan: Correct.

(01:41:26 PM) Joe: it would be good to have more security in place :)

(01:42:44 PM) Nathan: What is not secure about what we do?

(01:42:58 PM) Joe: the password. kept in plain text

(01:43:46 PM) Nathan: Its a very secure password, dont give it to people and you will be secured.

(01:44:16 PM) Joe: no. you dont understand. i'm saying: keeping password not encrypted or hashed on registryrocket.com part is not secure

(01:45:21 PM) Nathan: No one should know your billing password Joe. We see no reason to create a new password for that.

(01:45:39 PM) Nathan: I apologize.

(01:46:08 PM) Joe: no problem, but you still dont get it. if hackers hack into registryrocket, they have a database of domains and passwords for each domain. in plain text!

(01:46:31 PM) Nathan: Okay, you can change the password if you want.

(01:46:58 PM) Joe: but what if I change it today, and hackers hack into registryrocket 2 hours later?

(01:47:20 PM) Nathan: Then we will change the password. I dont understand how hackers would randomly guess your password Joe.

(01:47:59 PM) Joe: they dont need to randomly guess my password. my password is stored in registryrocket database in PLAIN TEXT

(01:48:45 PM) Nathan: Please show me where you can see it.

(01:49:10 PM) Joe: I cant!! I did not hack into registryrocket

(01:49:36 PM) Nathan: Okay, no hackers are getting into our database Joe. I do not understand how they would get your password.

(01:49:44 PM) Joe: but if someone WILL hack into registryrocket AND copy the entire database of users/passwords, then those passwords wont be encrypted or hashed, they will be in plain text

(01:52:20 PM) Nathan: Joe, no our passwords are encrypted with MD5.

(01:52:29 PM) Joe: impossible!!

(01:52:34 PM) Nathan: No one is going to hack into registryrocket.

(01:52:35 PM) Nathan: Okay.

(01:53:01 PM) Joe: ?? how can you encrypt password with MD5 and within 10 seconds from clicking "remind password", I got my password in my mailbox??

(01:53:54 PM) Nathan: Joe, our passwords are secured in a database. Please do not worry about us getting hacked. We have it under control.

(01:54:16 PM) Joe: you misleading me.

(01:54:40 PM) Joe: passwords CANNOT be properly encrypted in the database AND the same time you sending me password 5 seconds after I clicked "remind password"

(01:56:07 PM) Joe: even for very powerful computer it would take much more than 5 seconds to "decrypt" (or should rather say: crack) MD5-encrypted password

(02:00:35 PM) Joe: any news, Nathan?

(02:01:28 PM) Nathan: Our passwords are not MD5 encrypted, but they are encrypted and its done by Enom.

(02:03:06 PM) Joe: (01:52:20 PM) Nathan: Joe, no our passwords are encrypted with MD5.

(02:03:15 PM) Joe: whats "Enom" ?

(02:03:20 PM) Nathan: I apologize I made a mistake. Enom is who we sell our domains through.

(02:03:45 PM) Joe: ok, what do they use to encrypt passwords??

(02:03:55 PM) Nathan: I would not know.

(02:04:11 PM) Joe: I tell you: NOTHING

(02:04:54 PM) Joe: thats why most websites you CANNOT retrieve your old password, you can only reset it.

(02:07:22 PM) Nathan: What can I help you with Joe? Do you need the password to login?

(02:07:48 PM) Joe: I was just wondering how secure my vulnerable data is.

(02:09:22 PM) Nathan: We have a secure database, we have not been hacked in the past and I don't think we will be hacked anytime soon.

(02:09:41 PM) Joe: ok thanks.

Current Rating: 12345

I'm not sure what point you're trying to get across by posting that entire chat log aside from the fact that you're kind of a dick to the support person.

He wasn't super polite and didn't try to educate him, but I wouldn't call him a dick. There probably isn't any other way to communicate this message except via support.

I have worked as a "Chat Tech"/"Junior Administrator" --tier one support-- for a little over a year now for HostGator. I typically do phone calls rather than chats but the experience across each is similar. That said, none of this reflects the official endorsement or opinions of HostGator.com LLC.

The type of concern "joering2" has is obviously not something we handle at a first encounter level. It's not even a "support" issue in the normal sense. It's an infrastructure issue. Support chats and phone calls aren't going to end 15 minutes later with me having: designed, developed, tested, and deployed an entirely new password storing, reset, and re-send mechanism. This is true for any company that one has a widespread concern with regarding their policies and infrastructure. I may not approve of Microsoft's policy of forcing me through a "Customer Retention Specialist" when I wish to cancel my excess Xbox live subscription, but haranguing the poor girl I'm speaking to about company infrastructure and policy issues is a waste of her and my time.

That said, for HostGator, if you want to quickly reach someone regarding this type of concern, here's an email address: feedback@hostgator.com.

With regards to "registryrocket.com" I can assure you that it is and has been the bane of every tech's existence here at HostGator. Some explanation: HostGator used to, and still does, resell domains through eNom. When you visit "registryrocket.com" -- visit http://hostgator.com/domains in order to get a working session -- what you are seeing is eNom's idea of good default site design and then their idea of user experience. The user experience with this tool is... lacking. To put it this way: I've never had a call where a customer told me how easy and great purchasing a new domain through hostgator.com/domains was.

Moving on, this is not a system that we directly administer and control, so far as I know. It is a service hosted on and developed by eNom, as stated on their own page describing the product: "We take care of the website hosting, merchant processing and other eNom services." (http://www.enom.com/help/faq_rr.asp). This, the poor user interface for managing domains, and the general frustration of having to work with an external registrar rather than being in complete control, has led to us forming our own registrar: "LaunchPad." We're already registering new domains under this tool and getting a live web interface for it has been a huge request both from our customers and us lowly tech and sales staff. Here's to hoping the hiring push for development talent pays off :)

But there is: http://news.ycombinator.com/item?id=3474342

Tech support, especially tier 1 support, are there to follow procedures. No matter how fouled up those procedures are.

If the procedures need changing, you find the tier-1 support's management level. Going to the CTO/CEO if you must.

Since you seem off on the issue, it's quite possible to store a password as encrypted in the database and still be able to decrypt it.

I agree that MD5 isn't what they're using, as it isn't reversible encryption, or even encryption at all, but that ignores the fact that it's very possible to store a salted, encrypted password in a database, the salt on a separate file system, and still be able to send you your password in plain text.

For all intents and purposes, reversible encryption of passwords might as well be plain text.

Eh. There are ways to keep reversible encryption pretty secure, but while I agree with the notion, it makes a huge difference if the asshole on the other end of the phone keeps yelling about PLAIN TEXT passwords.

This was a nightmare. I was trying to get some urgent changes committed for a client that needed it this afternoon. Just about to commit around 14:00 est and Comcast takes a dive. Comcast support says they'll have it fixed by 5. Not great, I wanted to be done with work early on a Friday, but client is on the west coast so it will still be okay. Comcast comes up and ... shit, can't log in to the server. Try a few dozen variations of my password, then scour the web for an hour looking for hints if I'm locked out or something. Finally decide to check Dreamhoststatus.com. Fuckwits. Only sysadmin has access to panel to reset my password. He's sick and only checking IM sporadically. Finally get logged in and commit at 23:45pm est. Comcast+Dreamhost fucked me and my client real good today. Not much I can do about Comcast, but our migration off of dreamhost is getting pushed up to the top of the list.

Sounds like it's time for some process review.

Backup systems admin.

Key passwords available in a shared (but encrypted) repo.

Multiple data links -- tethering through your cell phone or heading to the local cafe or library in a pinch.

http://passpack.com/

Best password sharing/collaboration tool we've found.

Reply on news.ycombinator.com