Dropbox Cofounder & CTO Arash Ferdowsi responds to yesterday's bug
blog.dropbox.com
blog.dropbox.com
The statements should come from the top, but should probably only be written by founders if they're actually good at this kind of thing.
There are a lot of companies who are relatively good at crisis communications with affected parties. Network carriers are often good (at least when talking to other networking professionals), although they often try to restrict the spread of information via NDA.
There are some reasons to limit disclosure immediately after an incident (if you think the vulnerability still exists and might put you at risk, especially for a security threat and not a regular outage). You absolutely want to include "why this won't happen again" in your message (which Arash kind of did this time), but you also want to accept blame at an emotional level -- you can do this in ways which make you look perfectionist and hyper-professional, vs. weak.
I don't actually care that much about Dropbox security myself, but if anyone in the cloud computing space fucks up badly enough and frequently enough to make users distrustful of cloud resources, it makes life harder for everyone else in the space. That is a problem for me, and for other startup founders.
That's exactly how I felt after the AWS outage. There was a slew of (incorrect) media reports about why cloud computing was inherently bad for your business.
What does the US military do when it perpetrates a boneheaded security screwup that might or might not have compromised a small amount of data belonging to a few people?
(No question, Dropbox's response to this screwup is totally inadequate, but comparing it to how another organization responds when it has killed people is a bit silly.)
Shooting someone for speeding toward your checkpoint who turns out to have actually been deaf or in a hurry to rush a sick kid to the hospital is sad, really bad for everyone, should be avoided if at all possible, etc. There are also a lot of liability and cultural concerns. If you're willing/able to make a real apology for that, it should be really easy to do so for something comparatively minor.
I'd like the full access logs, including timestamps and IP addresses of every time my account was accessed in this timeframe. I've written security@dropbox.com about this, and am waiting to hear back.
"I am unable to provide log data at this time.
We're working around the clock to gather additional data. We will notify affected users if we detect any unusual logins or activity in their account. We are reviewing our logs that record password authentication events in accounts.
We have not been able to detect any relevant account activity for your account during the time period in question, so we believe that your account was unaffected by the bug."
There is no way dropbox would be able to explain to them what happened without scaring them silly.
And if nobody logged in, there can't possibly be any danger to their account.
Opposed to who? People who do know what it means and should be scared silly but aren't because they've been beaten into submission by breach after breach after breach.
IMO, this is reflective of a corporate culture that places testing and security on the back burner. And while some people may be OK sending their data to such a company, the rest of us might not be.
His comments always come off feeling like 'sorry bros, it's really not that bad, and we fixed it so no problemo' unlike Drew's comments which generally have a much more personal feel to them.
I personally am not interested in an apology or a gaveling tone. These people are not my personal friends, I don't have an emotional investment in whether they pretend to care about my feelings. I have an objective intrest in how they choose to act and the information they give me so that I can make my own informed choices.
Ideally I'd like to see the actual bug in code.
That makes it sound like it was primarily the fault of the developer who created the bug - things like this indicate an inadequate process or culture not simply a mistake at the developer level.
Edit: Does Dropbox have any testers or is all testing based on automated tests created by the development team?
I expect it wasn't obvious at all. I think that sharing the details with the community might help prevent others from making the same mistake.
I think you are assuming too much.
Or decide how to balance the risk-benefit equation. Perhaps this will inform my choices of which things to keep in my dropbox and which to keep elsewhere. For example, I might decide that my secrets are fine there, but it is inappropriate to store client secrets in Dropbox since the client has certain expectations around my respect for their privacy.
Dropbox Team,
For the duration of the event described in the post on your blog on Sunday I'd like the time of login and IP address of any authenticated sessions during the window. I'd also like to understand why a post from one of your customers linked to via hacker news was the only notification I received as one of your paying customers. If you know which user accounts we logged into during the event it seems rather straight forward that you would notify those impacted. It seems clear that had a 3rd party not brought this to light you'd have felt it unnecessary to notify your customers.
I look forward to your prompt response.
PS - because votes are not publicly visible.
We're working around the clock to gather additional data. We will notify affected users if we detect any unusual logins or activity in their account. We are reviewing our logs that record password authentication events in accounts. We have not been able to detect any relevant account activity for your account during the time period in question, so we believe that your account was unaffected by the bug.
Regards,
I'm going to change my behavior, the only things that go on Dropbox now will be completely public. They've completely lost my trust.
In the case of Dropbox, I would suggest PGP or Truecrypt anything sensitive and keep the keys locally or in another location that is completely unrelated to the box.
More details would have been nice too. Allowing anybody to log into anybody's account is a big deal, even if in the end a small percentage of people were likely affected. It's not like I couldn't access my account for a few hours or that the sync got messed up somehow.
Also, it'd be nice to know how the bug was discovered on Dropbox's side: did they realize it themselves or was it from nice people who found the problem?
Might be a huge lesson here to learn for other services to monitor their authentication system in both directions.
It's very disconcerting that they don't seem to be doing this.
It still shouldn't even be possible to deploy without some basic sanity tests. I'd put authentication very high on that list.
{
// looks good!
letThemIn();
}User Joe's account is logged into by attacker Tom. Tom sets his computer as one of Joe's "My Computers". Does what they did to clean up the problem invalidate this or does Joe have to log into his account, look at his list of "My Computers" and remove the ones he doesn't recognize manually to stop Tom's system from automatically syncing all his stuff until he does finally notice (which is likely never for most users, I'd assume)?
Probably worth checking your own sharing and connected device settings.
== Connected Computers and Devices https://www.dropbox.com/account#manage
== Sharing https://www.dropbox.com/share
The problem as discussed many times is that of security and usability, dropbox takes the usability route and for some users this is great for others not so much.
This might explain why Dropbox's poor communication of this security breach is the #winning move.
LastPass Disclosure Shows Why We Can't Have Nice Things
08 May 2011
A few days ago, LastPass announced they would be forcing their users to change their master passwords in response to what was essentially "something weird":
"We take a close look at our logs and try to explain every anomaly we see. Tuesday morning we saw a network traffic anomaly for a few minutes from one of our non-critical machines. These happen occasionally, and we typically identify them as an employee or an automated script.
In this case, we couldn't find that root cause. After delving into the anomaly we found a similar but smaller matching traffic anomaly from one of our databases in the opposite direction (more traffic was sent from the database compared to what was received on the server). Because we can't account for this anomaly either, we're going to be paranoid and assume the worst: that the data we stored in the database was somehow accessed."
LastPass acted exactly like we wish most companies would act: responsibly. And the media's response? Declaring LastPass "hacked" and "vulnerable", and placing them in the same category as Sony—who definitely were hacked—with sensationalist headlines like:
WARNING: Your Web Browser's Master Password May Have Been Stolen -- Change It Now LastPass Has Been Hacked And Asking Everyone To Change Their Master Passwords LastPass Hacked, Change of Master Password Urgent LastPass Is Hacked – Change Your Master Password, But Don't Panic Should the LastPass, Sony hacks make you fear storing data in the cloud?
I'm not trying to defend him but I am genuinely curious what other people think his response was missing.
2. The error is presented like casual news. Sort of downplaying the incident and not acknowledging that they screwed up. If the accounts were freely accessible for 4 odd hours, it is pretty serious.
3. No actual details of bug introduced were provided.
4. The communication was done on blog, and emails were not send. Again, it is an attempt to downplay the incident.
Seems reasonable to me.
That doesn't sound reasonable in the least bit.
Ya know, just in case you want to upvote it.
I understand the challenge that the Dropbox team finds themselves in here, but I'd very much like more details as to exactly what happened and what was possible for 4+ hours. I'm less interested in they're eventual report of the bad stuff they think did or didn't happen during that time.
That said, I can't help but feeling misdirected. I mean, obviously the cap on the number of compromised accounts is relevant, but I think more relevant is the fact that 100% of the accounts were completely insecure for hours.
Misdirected, because as a user I don't care at all how many accounts were actually compromised. This isn't a no-harm-no-foul incident. It's an enormous breach of trust that causes me to completely rethink what I'd be willing to do with their service.
How does one accidentally set the failure mode of any piece of authentication code to escalate privileges instead of denying them? Even when I was first learning to code web apps I never found myself accidentally accepting any password.
It's shocking (to me, at least) because it seems like such a beginner mistake.
It's very doubtful something like Dropbox is as simple as
if(user.hashedPassword == sha1(salt + enteredPassword)) {
letUserIn();
} else {
error();
}
if(user.hashedPassword == sha1(salt + enteredPassword)) ; {
letUserIn();
}
Production
----------
@authenticate
def foobar():
....
-----------
#@authenticate ( Because developers are lazy, while testing)
def foobar():
....
In my case though it included "we noticed some potentially suspicious activity during the period", which for me was linking the desktop app. I don't know if everyone will be receiving a note, or just those who fit the potentially-suspicious profile. A quicker notification would have been nice, but I appreciate that they're also obviously looking for any aberrant behaviour.
"We have strict policy and technical access controls that
prohibit employee access except in these rare circumstances."
That's why you shouldn't trust statements like: "Theoretically we can read your data, but we make sure only you can access it". If there's the theoretical potential for abuse it will happen sooner or later. Either deliberately or not.
I am happy with not putting sensitive files on Dropbox, and encrypted the few I do.
As long as this average doesn't spike I am very happy with Dropbox overall. They have enabled me to do so much and to work better.
On top of not letting something like this happen (where were their tests?), they should be diligent in talking about something like this immediately so that people know they're not being lied to, ever, and that they can trust the service with their data. By not doing so, we have to wonder about how many other bugs do we not know/hear about that might affect our accounts too?
Better if it tests the production site directly, in addition to testing in a test environment.