Zappos.com customer database compromised | Hacker News Reader

Zappos.com customer database compromised

zappos.com

199 pointsclamstar89 comments

LastPass FTW! The attacker will reverse my password just to find a bunch of unusable bits :). What would be even cooler is an API on top of LastPass that sites like Zappos could hook into to force a behind-the-scenes change of passwords, similar to revoking a compromised certificate. Essentially, since there is some lead time after the breach is discovered and before the attacker manages to crack the long, random passwords, their efforts would be futile by the time they are done since all LastPass passwords would have already been changed.

Or we could just stop using passwords everywhere and not have this problem again. Anybody? Anybody?

Disclosure: I have no affiliation with LastPass beyond being a satisfied user.

> What would be even cooler is an API on top of LastPass that sites like Zappos could hook into to force a behind-the-scenes change of passwords

How would Lastpass protect against an attacker masquerading as the third party website? (Especially considering this feature would be used when a website finds itself compromised.)

Maybe an API is an overkill in this case. Instead, a simple web service with a twist: Zappos has a private key and LastPass has the corresponding public key. Now, if Zappos.com is compromised and the breached is discovered and fixed, their CEO/CTO/head security guy grabs the private key and authenticates to LastPass, telling them that he is in fact who he says he is, and finally triggers the massive automatic password reset. Obviously, this will not work if the private key is compromised, but then again, our whole web security paradigm is "trust that the website owner knows what s/he is doing", so this is already a step up.

Or, as I mentioned, let's do away with passwords. Anyone can have your public key so long as your private key stays private.

Well, lastpass doesn't store the passwords on its servers in a way that they could just change. From my understanding the database is only decrypted on the client machines when the master password is entered.

Still, the idea of a service for handling this makes sense. Rather than one based on a single vendor, a simple API for querying compromised domains would handle it. Then the lastpass extension can call that api for a list of the user's domains and see if anything needs to be changed. Being more general (just giving out information about recently compromised sites) also seems more useful, in that people would do a lot of different things with it.

Except when LastPass was compromised last year...

FFS! It wasn't compromised, not remotely. The incident last year is what convinced me I could trust last pass.

Well they said that their database was compromised and they were not sure what was accessed.

So I stopped using them after that incident.

It was a while ago I don't remember the particulars, but I do remember they said they were not sure if someone stole everyones password so everyone should change their master password to be safe. So I deleted my account to be safer.

For that reason, I find the 1Password model more suited to my tastes. Using Dropbox to sync, it works just as nicely and I'm not beholden to a third party central database (LastPass).

  Well they said that their database was compromised

No they didn't.

  I don't remember the particulars

Then why do you make such explicit claims about what happened? They spotted a traffic anomaly on their network and went into complete paranoid mode. It is completely unknown, even to them, whether someone unauthorized accessed their database or whether they just couldn't account for some traffic on their internal network.

I don't know anyone else that monitors the traffic on their network to detect unauthorized access and I know many companies that don't. That's already a huge plus and it makes me trust them with security in general all the more.

So what? As long as your master password was strong, your data was safe.

There was also the XSS flaw in Feb last year that allowed an attacker to retrieve your email address, your password reminder, the list of sites you log into and the history of your logins, including which sites you logged into, the time and dates you logged into them, and the IP addresses you logged in from.

https://grepular.com/LastPass_Vulnerability_Exposes_Account_...

Their reaction to this flaw was exemplary though, and LastPass is a lot more secure now because of it.

I used to have three different passwords of varying complexity that I shared across sites.

When Gizmodo's database was compromised and I didn't know which password I used there, I decided to stop using the same set of passwords everyone and started generating and storing my passwords using 1Password. It's a little annoying to use on my iPhone (particularly having to type my long master password on the soft keyboard), but it's dead simple to use on the desktop and I recommend it to everyone. I still have some sites that use my old passwords, but 1Password's Smart Folders let me search my passwords for those and I plan on changing those today.

(I haven't used LastPass so I can't comment on how it compares to 1Password)

This is exactly what I do and I've switch friends and family over as well.

Whenever they bring up the perceived inconvenience (which goes down on the desktop with practice) I simply remind them how much time they will waste if one of their accounts is compromised.

Sure their foursquare (or pick another random service that doesn't hold EXTREMELY important data) account isn't that important but when it uses their Gmail address and has the same password they are just begging for trouble.

Also this gets them out of logging on to their Gmail and Facebook accounts from public computers. They still don't fully understand the possible problems but at least now it is such an inconvenience they just use their own devices.

Zappos developer here. I'll answer any questions that I legally can or help get customer problems passed onto people that can help.

Was the password hash generated using bcrypt?

Plaintext passwords never touch our database. Expiring everyone's passwords was a security precaution given the fact that our non financial customer data was compromised in the first place. I can't comment on what lib or algorithm we use to encrypt our passwords since I don't work on that team.

Hi. I'm customer outside of US and I received the email, went to site to reset my password and "We are so sorry – we are currently not accepting international traffic" - WTF? (sorry, but there is your logic?)

International traffic will be re-enabled in the near future.

Why was international traffic disabled?

Just a precaution while we asses and deal with this. Zappos doesn't ship internationally so we hope this isn't affecting many customers. But to those that are, we apologize. As soon as we can we'll re-enable traffic from outside the US.

also.... "assess" not "asses"

I for one would love Zappos to ship internationally, and your owners at Amazon already do. I know you can't comment, but please do what you can to push for selling to the other 6.7 billion of us.

It'd be cool if we could ship internationally. What would be even cooler is if we could inspire entrepeneurs in other countries to tackle it!

I'm in Chile as part of Start-Up Chile. What exactly do you need? PS - I am also a US customer. A block on international traffic effects more than just international customers.

Manpacks.com is a great example of a company that started up to fill the void left when Zappos stopped shipping to Canada.

Ah, I understand now. You mean you want companies to just replicate Zappos outside the US. Any chance that there could be a "Zappos API" where we could while label Zappos, and pick-up the goods at the Zappos warehouse for foreign shipment?

try http://fifty one.com. They seem to be still growing, but facilitate international payments and shipping for a bunch of companies.

Javari (javari.co.uk) is another shoe store owned by Amazon and they do ship internationally (free within EU and Ireland)

I think they developed Javari in most of their European markets and logistics are handled by Amazon themselves.

Zappos is operated completely separately from Amazon so in order to expand internationally they would have to roll their own operations internationnaly.

Good job on the fast response! This is the first time I've heard about a security breach from a company before seeing the dump on pastebin

and to jump on social media to answer questions. the future of professionalism.

How was the compromise discovered?

no clue

As a developer who fears these kinds of attacks on my own sites, is there anything you are able/allowed to reveal regarding how the attack happened, how it was discovered, and/or how it could be prevented?

Not at this time. Sorry.

Do you know what hash was used, if the passwords were salted and if so, if the salt is secure?

I'm not on the team that handles passwords so can't comment. Sorry.

Hmmm, please ask the team that handles passwords and let us know.

What do you mean by "the salt is secure"? Hidden in code files vs. stored next to the hashed password?

I apologize if my question was unclear; that's almost certainly because of a lack of expertise on my side.

On one end of the spectrum, I envision the same salt used for every user, allowing for the easy and effective creation of rainbow tables. On the other end, I envision unique salts with many bits of entropy for each user, making rainbow tables technologically infeasible.

The email did not mention order history. Do you know if our personal order history was among the items compromised?

Not sure. Sorry.

There really is no good news in this type of situation, but only the data items mentioned by Tony in the email above were compromised.

i'm picturing the blackmail note:

"dear bestnameever, i know about those high heels you bought, and i happen to know you don't have a girlfriend. $1000 in unmarked bills or we tell your father you're a cross-dresser."

Or the incredibly geeky wife who suspects her husband is showering the hot secretary with shoes and handbags, and confirms it by poring over the breached data.

Zappos developer here. I'll answer any questions that I legally can or help get customer problems passed onto people that can help.

Can you provide any further information that would be of interest to HN readers? If not, why do you bother posting this?

Bravo for being on top of things and answering questions on HN!

Good job on not storing or sending clear text passwords. However, as others have indicated, we would like to know more about the hashing method used.

As a side note, I was horrified to discover that Hertz sends passwords (as part of password recovery) in the clear. For those using Hertz, you should take the appropriate precautions.

I'm guessing posting anything about our password hashing will be frowned upon due to the context of this question.

What are the best practices you want to see a website use when storing your password?

bcrypt http://en.wikipedia.org/wiki/Bcrypt

+1 for bcrypt - "ordinary" hashing algorithms were made to compute as fast as they can, which is exactly the opposite of what you will want for your system. Rainbow tables are so quick and easy to make - IIRC it currently takes only some hours to compute all MD5 hashes for passwords up to 8 characters long on a system with some good graphic cards. What you want is an algorithm which takes an up-to-date system some 10-100ms to compute a hash - bcrypt is configurable in its complexity (time to compute hash), and you should adapt the parameters every 1-2 years to increase the complexity.

Security through obscurity should not used. Just saying. IMO, revealing the method used should not become an issue just like the reason why the more trusted crypto algorithms are publicly posted.

That said, bcrypt and a time/attempt limited lockout should go a good ways in securing your site.

> I'm guessing posting anything about our password hashing will be frowned upon due to the context of this question.

The only way it could be harmful to disclose the hashing method is if you're using an insufficient one.

MD* - No, SHA* - No, Bcrypt - Yes!, Scrypt - Not yet (PDI). Make sure you also calibrate the work factor for Bcrypt, too. Then, write a blog entry about your findings.

Salt. Seriously. A big, long, gnarly-looking salt. Preferably a unique salt per user. Really, even just that is sufficient, even if stored right next to the hash. It means doing a bruteforce/dictionary attack one user at a time rather than one bruteforce/dictionary attack of all the users at once (static salt) or just googling the hash (unsalted hash).

Also, are these passwords encrypted or hashed? Those two are miles away from each other and you guys are using both words nearly interchangeably. If encrypted, where is the key? Was it compromised?

Salt is no longer sufficient.

What does it mean if you are a Zappos customer who did not get an email?

All Zappos customers will be receiving the email linked to in this thread. If you haven't it might be in your spam folder or it might still be queued to go out. The link above is the same as the email contents.

I don't seem to be receiving the password reset email. I do use an email with a plus ("foo+bar@example.com") if that has any effect.

Most likely your password reset email is in the queue to go out. Emails are slow to go out due to the massive volume of outgoing email we are trying to send out.

Obviously in this kind of situation we (Zappos customers like myself) need to change any re-used passwords since the stolen unsalted hashes :( can be cracked. However, I have no idea which of several passwords I used at Zappos! I would normally just try logging in with each of them, but since you've reset all passwords, it looks like I won't be able to. Is there any chance of helping with this? I need to make sure it wasn't a password I use on any important sites (or derived from such a password).

I'm looking for the data dump right now, in case it was posted publicly--that's probably the only way I'll be able to answer my question since I doubt Zappos will cooperate :(

As someone who was just bit by the Stratfor data loss, this is the second month in a row. Fortunately my Stratfor password was worthless, but I had my credit card stolen and used to pay for video games. And now my email and street address are public information.

+1 for not storing clear text passwords.

I like the tone of the blog & how forthright they have been with dealing with the issue.

Agree. So many companies don't act like grown-ups and just try to cover up the problem.

Still, it's going to be pretty tough getting your average customer back who hears they've been "hacked" and are afraid to create a new password. Not to mention the average customer's password is probably the same password across facebook, gmail, etc.

Absolutely. The biggest risk is the shared password part. It is surprising people still do it.

I am surprised that some of the big eCommerce companies still mail back the password in clear text. Just plain stupid.

Sharing passwords will end when I don't have to remember one for every random website ever.

Zappos is always a class act. I have about 3X the shoes I otherwise would have as a result of their customer service.

> +1 for not storing clear text passwords.

That shouldn't need a +1.

It shouldn't, but it's shocking how many companies don't encrypt passwords before storing them in the db.

Considering that 90% of success is showing up, and the next 9% is avoiding obvious failure paths, Zappos is doing pretty well here.

Lots of room for improvement above and beyond these two points, sure, but at least they're not falling victim to the classic blunders.

Disallowing international sales means they'll probably also avoid getting involved in a land war in Asia.

Now if I can just find my iocane powder...

While they do get "+1" for this, they haven't provided any further details of what exactly they did with the passwords. Did they use a salt? Was the hashing algorithm MD5, bcrypt, or something else? If they used MD5 with no salt, your password may not be much more secure than a clear text password unless it's particularly complex.

Page gives me : "We are so sorry – we are currently not accepting international traffic. If you have any questions please email us at help@zappos.com"

Anyone could paste/screenshot/... what there is to see ?

Subject: Information on the Zappos.com site - please create a new password

First, the bad news:

We are writing to let you know that there may have been illegal and unauthorized access to some of your customer account information on Zappos.com, including one or more of the following: your name, e-mailaddress, billing and shipping addresses, phone number, the last four digits of your credit card number (the standard information you find on receipts), and/or your cryptographically scrambled password (but not your actual password).

THE BETTER NEWS:

The database that stores your critical credit card and other payment data was NOT affected or accessed.

SECURITY PRECAUTIONS:

For your protection and to prevent unauthorized access, we have expired and reset your password so you can create a new password. Please follow the instructions below to create a new password.

We also recommend that you change your password on any other web site where you use the same or a similar password. As always, please remember that Zappos.com will never ask you for personal or account information in an e-mail. Please exercise caution if you receive any emails or phone calls that ask for personal information or direct you to a web site where you are asked to provide personal information.

PLEASE CREATE A NEW PASSWORD:

We have expired and reset your password so you can create a new password. Please create a new password by visiting Zappos.com and clicking on the "Create a New Password" link in the upper right corner of the web site and follow the steps from there.

We sincerely apologize for any inconvenience this may cause. If you have any additional questions about this process, please email us at passwordchange@zappos.com

You can check the URL http://viewtext.org/article?url=http://www.zappos.com/passwo... in case content changes/updates in that page.

Adding the one really interesting, that, the way they communicate in house: http://viewtext.org/article?url=http://blogs.zappos.com/secu...

So: "cryptographically scrambled" -- do we believe they use a good hash, and salt? Or... not?

I didn't get this notice so that means my information wasn't compromised? Wouldn't bet on it.

I didn't receive the email yet either. Hmm... just changed my password just in case.

Yeah, I think they're sending the email out in separate blasts. This is one email they don't want ISPs blocking IPs on because it looks like spam.

As a rule, anytime something like this happens, you should change your password (and any other place you use the same password or a variation of it). That being said, we are bursting the emails (as I believe Dylan commented already). Spam filters being what they are, it's possible it may be in there, so I would suggest you change your password.

Zappos sister site 6pm.com was compromised, too.

Correction, it was not a separate compromise, they share the same database (since they share the same stock, etc.)

My thanks to Zappos for that email. It was enough for me to give my wife necessary suggestions to secure her associated accounts without alarming her.

It is probably worthwhile in these situations to provide basic implication info for laymen, i.e. implications of "your cryptographically scrambled password."

I've been having issues with Zappos for a couple days. I called up support yesterday and they said they were "upgrading the website and had bugs they were trying to get fixed." Not sure if this is related or just a coincidence.

Probably coincidence. Companies that expect a lot of Christmas traffic minimize changes from Thanksgiving to Christmas, and web retail gets more traffic during business hours in America, so I expect that this weekend and last weekend saw a lot of code deployments, and so things are more likely to be broken specifically right now than pretty much any other time of the year.

Good thing they didn't store passwords in clear-text!

That doesn't mean they're not using a bad (i.e., fast) hashing algorithm.

Or that they're not using reversible encryption.

a lot of poor hash/encryption implementations are close enough to plaintext.

I didn't get an email, but upon logging in - my password was reset and an email sent with further instructions.

FWIW, I just got a similar email from 6pm.com (It's a Zappos Affiliate) ..

Reply on news.ycombinator.com