Hacker News Reader

Availability is a core part of Security (not just Confidentiality or Integrity). In that respect, I think DO is doing the right thing - they eventually scrub all data, but leave a small window to recover from mistakes.

Of course, it depends on the use-case and customer priorities, but in my opinion, availability is a much bigger concern for most DO customers than confidentiality. If customers are deeply concerned about the latter, I would say they are less likely to choose a virtual platform in the first place.

I think this could be corrected with a small note (backup data will be scrubbed within 24 hours. If you wish to remove this data sooner, click here).

Having experienced making a mistake and removing the wrong virtual host (albeit on Linode), I really appreciate the fact that it could be recovered. And yes, I have external backups and could have recovered it otherwise, but it was much easier to have an 'undo destroy virtual host' button (so to speak) than to kick off my restore process.

I hope Digital Ocean and other cloud providers don't change the default behaviour. It will hurt many more customers than it would help in my opinion.

I agree with you temp images are good however that check box text is not phrased right it should either say that it takes up to 48h to happen or specify that it only happens on the hypervisor and give you the option to not make temporary backups as well.

Your approach seems very sensible and the data you provide is in support of this. I do hope you stick with this, and more importantly, your data drive meta-methodology, and not let an HN post who don't face the use cases etc. drive you to abandon this.

We love to hear feedback from customers because it allows us to refine the product. From our internal metrics we know that providing an automatic snapshot recovery function is essential given the number of times its been implemented but we can also understand this customers issue with data being destroyed.

The data is destroyed securely, but not as immediately as they would have liked since the default behavior is to preserve a snapshot until it automatically expires.

These discussions are great for us because they allow us to refine the user experience and ensure that we can provide a great service. As we've been doing a tremendous amount of work on the backend code this is a great time for this customers concerns to be discussed as we are knee deep in large rewrites at the moment and its the perfect time to address this, rather than deploy to production and then have to go back and retool.

Given that we've been working on the secure destroy all the time having that be an option doesn't make sense and instead we can use that UX/UI real-estate to expose the temporary snapshot issue but again would love to hear more feedback on whether that's necessary.

We would of course love to please every customer but then we would have a platform with 1,000 features so sometimes you have to politely decline a request because it doesn't fit in with the general use cases, but it doesn't mean that we shouldn't engage in a discussion as that is always helpful.

Even if action isn't immediately taken it often leads to us asking numerous questions that are always great that can lead to new product and feature ideas that may not be tied to the original question in mind but still can move the product forward in other areas.

Thanks!

I agree. I also think that the poster could have talked with a DO rep. before making this public. It seems like the intention was to "expose" DO. I could be wrong though.

Something doesn't add up. The user who wrote the report did not request that the destroyed droplet be recovered. The existence of lingering data was disclosed by traces which appeared in a brand new droplet. The only reason I can come up with for why that would ever happen is that you are re-using droplets, perhaps because erasing and resetting a few files is cheaper and faster than provisioning a brand new instance. If this is indeed the case, then what you are suggesting -- namely, that what looks like sloppiness is in fact conscious, customer-driven product design -- is completely false, because it means that the droplet recovery feature is available only in those cases where the customer did not create a new droplet after destroying the droplet she would now like to recover. I guess that this is why you were "able to recover almost everyone's droplets" after an attack, but not everyone's. The "temporary snapshot" feature is not actually a feature, or even a "mechanism" -- it's a bug, just one with a side-effect is occasionally positive.

Sorry that's on me for not explaining better:

I created a droplet and then destroyed it with the 'Scrub Data' checkbox enabled, and was surprised when I noticed it had been turned into a "temporary snapshot" rather than eliminated entirely in a secure, irrecoverable manner as the text around that UI element would have suggested.

I then went ahead, to prove a point, in restoring said snapshot onto an instance called 'test' and what you are seeing in terms of lingering data is from 'test' as very primitive proof that the snapshot did indeed occur and 'Scrub Data' doesn't behave how I think it should.

tl;dr - raiyu is being perfectly forthcoming and DigitalOcean is not reusing droplets to avoid erase/reset, we're just quibbling over the semantics of whether the 'Scrub Data' box should have a safety net or not, and whether the presence of a safety net could be deemed a security issue.

I read the blog quickly and didn't understand that you restored the snapshot either. I think that's something you should highlight better, because it completely diminish the level of the issue.

Agreed. It took me a while to figure out and it basically makes your post moot, from what I can tell. I thought you made a brand new droplet and found your old data on it.

Moisey - Hat tip definitely due, you've been nothing short of responsive about the concerns raised and I can completely understand what led you down this train of thought, although I continue to think it presents an unwelcome surprise to customers thinking about their data security.

Destroyed should mean destroyed and if the outcome is a UX/UI change to remove the 'Scrub Data' (making it default) box and replace with a 'Temporary Snapshot' box (along with accompanying tooltip for explanation) then I think that would be a perfectly great way to assuage my concerns, and a positive development to the DigitalOcean platform.

Obviously there remains a question of how best to tackle this from an API perspective, perhaps you can consider adding a new parameter to /droplets/:id/destroy along the lines of 'temporary_snapshot' (bool) or similar, with a well-documented default behaviour if not passed in.

Agreed, this highlights a larger issue that we've been dealing with as we've grown to over 100,000 customers and the platform has evolved.

When you have a singular product with a very narrow focus, say Instagram, it allows you to really strip away everything besides the core functionality.

When building a platform often you start off with something simple and elegant and as the number of customers grow you quickly begin to realize that as a platform you need to provide more customizability than first intended.

With that in mind we've been working with our customer support team to understand what are the most frequently asked questions and began to automate that directly into the support system as well as provide a more relevant and up to date help guide for the platform. Both inside the control panel directly but also searchable.

Of course one of the thoughts was to make all of the help documentation public on github and then also field requests from the community to help rewrite any documentation that wasn't clear.

Sometimes when you are engineering a product you take for granted all of the knowledge that you have and something that is obvious is entirely missed.

So that would be the idea behind asking our community to help us update the documentation since they come in with fresh eyes and can spot things that we can't see because for us it's been a continuous spectrum of development rather than something that we just walked into at some point in time.

Thanks!

You mentioned in a comment above an occurrence where someone's DO account was compromised leading to many droplets being trashed but that the account holder was able to recover because of the waiting period. If you add a flag to bypass the temporary snapshot would not this mean the hacker could have succeeded in their attempt to wipe out all the droplets?

I actually prefer the enforced cooling off period for destroying droplets. (As long as the UI/API docs are cleaned up a little to communicate it better.)

Replacing one checkbox with another is not solution, since it will remain optional for customer and thus can easily be missed when submitting the form. For such sensitive topic there must be explicit confirmation of user intents.

I see where your coming from a lot of people don't seem to understand the meaning of permanently and irrecoverably deleted.

My suggestion is to remove the secure delete flag/option there's little reason not to securely delete things and make the default behavior on both websites and the API as delete with 24h-48h backups.

However add a flag/option that does permanently and irrecoverably delete the droplet without making any backups.

This flag/option should require active input from the user so there are no excuses or misunderstandings once checked it's gone for good.

Semi offtopic... I love your interface and service. However, IPv6 is a requirement and you don't offer it. It's been promised for years now, but you leave your customers in the dark. Either say no, like aws, or yes. Please don't keep dicking with us.

There was just some news on that front, actually. There should be an IPv6 beta in Singapore in a few weeks.

http://digitalocean.uservoice.com/forums/136585-digital-ocea...

Hope so this is a programming issue.

But I think DigitalOcean is doing the right thing for customers.

How did you solve the original performance problem with scrubbing by default?

You've responded well -- all you can hope is that people understand.