Comment by sneak | Hacker News Reader

Comment by sneak | Hacker News Reader

> At no time was customer data "leaked" between accounts. This would require that a user explicitly not scrub their volume after destroying their server; in this instance data would be recoverable and should be considered not sensitive.

For fuck's sake, now you're just lying.

Not scrubbing has been the default - a user doesn't have to "explicitly not scrub". A simple unadorned "destroy" api command creates the circumstance in which the data is not destroyed, but made available to others.

If no customer data leaked between accounts, how was I able to read someone else's stack traces[1], web logs[2], and customer tokens[3] on a freshly provisioned VM?

What follows is evidence to directly support the claim that you're lying through your teeth to save face after having been caught being grossly irresponsible with your customers' data.

Please start acting like professionals.

[1] http://i.imgur.com/TMp2kdf.png

[2] http://i.imgur.com/WLv2qSE.png

[3] http://i.imgur.com/fJOxRN9.png

Agreed. This is a lie. Plain and simple. Even of you don't know any of the particulars about this issue, the post just contradicts itself. This is a textbook example of how not to own up to a security issue - if anything, this response makes the entire thing worse because you realise the actual problem (their approach) isn't being fixed, only this specific manifestation of the issue.

> you're just lying

You are aware you are talking to more than one person at DigitalOcean, right? Their post has contradictions in it because the person that wrote it isn't the same person who is implementing the fix. Clearly they know they fucked up, are fixing the fuck up, and are trying to implement damage control. The people doing the fix for the fuckup are fixing things because they agree with you. The people writing the copy for the damage control don't agree with you. It's a clear case of cognitive dissonance, which isn't surprising given there is more than one person involved over there.

Individuals already have cognitive dissonance to begin with, it just gets worse when more people are involved. A company without cognitive dissonance would have never implemented this to begin with. Celebrate the fact you got them to change it and move on. Fighting them with blaming statements is just going to make the people who actually fixed it for you sad and disappointed. That's not what any of us want.

And, you are right, someone over there is just lying and someone else over there in charge should take them aside and explain how it's not cool.

Engineers make this mistake all the time, but it's important to remember that for everyone outside our field, the fact that you can explain why something is happening does not make it ok. Things that are wrong with this communication:

* There are contradictions in claim

* There are misstatements of fact

* They are offering a poorly qualified apology

> Fighting them with blaming statements is just going to make the people who actually fixed it for you sad and disappointed.

Disagree 100%. If the management team can't get their head around the "right" decisions from an engineering perspective, the engineers are in for a world of sadness and disappointment. The management at DO need to hear every one of the criticisms being leveled here and on their own communications channels.

I will agree not to blame corporations for the actions of their employees the moment those employees are individually liable for their actions.

Corporations always speak with one voice. If that voice says something wrong or inconsistent, it is not the public's problem, it is the corporation's. No excuses. Ever.

Open Source forever, baby!

That has nothing to do with the conversation.

Seeing how I own a parent of the comment above I think I can safely state it does. Open Source software has a set of methodologies that provide trust and transparency. Accountability is but one side effect of such a system. It's not a stretch to see that extended to the way corporations deal with content generation.

See Joel at Buffer's post on open salaries for reference.

I read that they acknowledge they made the default to not scrub...

"As a result, we switched the default mode away from scrubbing to improve performance, given that customers would have complete control over this action themselves."

Also, I think they presume a leak to be qualified by the data being sensitive which would not be leaked had it been scrubbed, for which they provide an option. This has to be assumed, otherwise one would never need to scrub.

Still, certainly looks to be a pretty big mistake in communications.

You're being ridiculous. They made a conscious decision to increase performance and the lifetime of their SSD drives by turning `scrub` off by default. If you enjoy DigitalOcean's prices (starting at $5), there must be some balance and optimizations to continue to run a business.

In the control panel, its quite clear, and easy to simply check scrub on. I agree that this change on the API front should have been articulated. However, Moisey acknowledge their mistake, let's move past it.

No. It's not ridiculous. What's ridiculous is that they're even trying to claim no data was leaked when it demonstrably was.

There are many ways to securely scrub an SSD without subjecting it to a full write. They were outlined in the original thread.

Customers should not be able to access other customers' data under any circumstances.

This has nothing to do with scrubbing, or SSDs. This has to do with them providing my data to third parties without my consent.

They've leaked customer data between VMs. Their claims that this isn't true are false. Look at the screenshots!

No one would intentionally make the choice of having their data shared with other users, and if that is a business model they rely upon they need to find another.

This is a profound, never-ever-should-happen gap in data security, and it is yet another instance where DigitalOcean comes out looking remarkably amateurish.

Conversationally, I would have expected their hypervisor to have been doing thin provisioning: That until you write data to a block the block is 0s (having no provisioning on actual storage). And if you write 0s, that too isn't actually provisioned on real storage. And when you write actual data, well that is what the block now contains.

>No one would intentionally make the choice of having their data shared with other users, and if that is a business model they rely upon they need to find another.

Unless they do not care if their data is shared, and would be willing to let it be shared for a reduced cost.

If there were a box that said "share your data" that you had to click, I cannot imagine the users who would click it. Further no one was ever told that the value proposition of DigitalOcean relies upon a complete and utter lack of data integrity.

That is a dishonest angle. It is not in any way how this has been sold. Further you don't get a discount for choosing not to scrub, but instead essentially get tricked into it by the magical law of defaults.

I do not mean to say that it was justified in this particular case. I was responding to the idea that all products must be secure, even for users who do not want to pay for the added security. Also, it is not so much that I think a service should offer a discount for handling your data insecurely (although in this particular case, the insecure handling is actually cheaper on a per user basis). Rather, there is a place in the market for products that are not secure, and therefore do not have to pay to develop and maintain the security aspects of the service.

And yet all the other major VPS/cloud server providers have engineered solutions that do not suffer from this issue (regardless of physical storage medium). Specifically, Amazon, Linode, and Rackspace all have automatic mitigations (in varying forms) that prevent this type of leak.

Reply on news.ycombinator.com