DigitalHax – Allows you to recover data from "Destroyed" Digital Ocean VM
github.com
github.com
We've now default scrub_data to ON for both web interface and API as we look at making this process permanent. Additionally, we've re-engineered the way we're provisioning disks and access to previously written data is no longer possible.
We've taken all steps in favor of security currently and will build a permanent solution that favors security and caution moving forward.
Like I've said before, I care very much less about the existence of problems than I care about the timely and appropriate response to them.
A question: is the lapse here going to trigger any bigger-picture analysis of your security practices?
You guys really need to become better at communicating with your customers when I can look at the front page of HN one day and see some issue with your services, DO people commenting and no mail in my inbox.
The priority should be to alert customers there is a problem, and most importantly to fix the problem.
And sending a mail a week or a few days later is really not okay, a rapid response on your end to notify us is needed if we are going to be able to quickly take necessary precautions.
Edit: Here's the code. https://github.com/gregimba/DigitalHax/pull/1
Read a lot of other people's code to see how stuff is done, it helps a lot.
At one point in my career, I did electronic discovery and electronic forensics for a bankruptcy trustee as part of their fraud investigation process. One of the central principles is that you want to take every step you can to prevent changes to the examination target. The best way to do this is to mount the volume read-only from a boot CD or separate partition from the examination target.
The Digital Ocean documentation says that you can request that your Droplet be switched to a recovery ISO, but you have to make the request through support. I don't know if this is a technical limitation or a policy decision, but it ends up being a good mitigator against a mass examination of data using these techniques. You can see more details about how to request this under the section section "Attempt Recovery with a Recovery ISO":
https://www.digitalocean.com/community/articles/how-to-recov...
You'll want to complete the network setup, but when it comes time to mount your filesystem, use the `-o ro` flag with mount
mount /dev/vda /mnt -o ro
So, don't take it as the maximum damage one can get.
update: finished in around 12 minutes. out.txt is around 10gb.
update: out.txt is around 54 million lines from wc -l out.txt. I'm using less with command [line number]G to poke around. I have an NYC1 droplet, and there's a lot of junk not mine.. text in other languages and python which i don't use
May I ask, which droplet type were you running dd on? Micro?
If I choose not to use that (and I never have on any of the hundreds of machines I've created and later torn down) it's because there is nothing of any sensitivity on them. If someone wants to resurrect gigabytes of entirely boring and transient log data from what I was last doing, they're welcome to!
I can only really see this being a concern for people who were storing sensitive information on a cloud instance which they then removed and chose NOT to scrub. In which case, they already have larger issues than this one. "Problem with user, not with cloud."
What you're saying is that insecure defaults are OK, so long as they're obvious. The problem is that things are rarely obvious to 100% of the people, 100% of the time. A company the size of Digital Ocean has enough customers that if even 5% of their customers misinterpreted this option, a "significant" number of people would be affected.
Consider, for example, a user who sees the destroy page and assumes that "scrub" data is some extra, optional precaution, because "destroy" can't possibly mean "erased but recoverable". I mean, it says destroy, right?
Just a few days ago, someone posted a link to a blog post titled Toyota Manufacturing Principles. One of the principles mentioned in that post was poka-yoke[1]; otherwise known as mistake-proofing. A tenant so important that Toyota -- one of the most successful industrial companies in the world -- has made it a core principle.
One of my business partners has a favorite catch phrase: it's never a problem until it becomes a problem. It's his way of pointing out that just because something hasn't happened to you yet doesn't mean it won't be a problem if it does. Speaking as someone who has made their fair share of mistakes, I'd caution you to consider that advice carefully.
As I said at the time, "that's a stupid option and a user should never be given that choice". So, I see where you're coming from, your point is well made. :)
You're seeing those comments on that blog right? http://i.imgur.com/f90Nx0V.png
I don't think everyone will be happy until you make scrubbing the default. Or I guess those direct reads with the "dd" command stop yielding data from previous VM instances, which it does kinda sound like DO is preparing to do. If scrubbing isn't going to be default, I'm kinda curious what DO will be doing to ensure clean VMs.
Forget to check that box? Oh well, better hope the next droplet doesn't go and read your data.
Moderately competent developer doesn't realize the implications of not checking that box? Oh well, better hope that developer didn't have too much sensitive data on the droplet.
Etc etc. Security is the big area where the default should be to err on the side of caution - often removing choices that are simply too dangerous (when, for example, the tradeoff is a tiny amount of performance gain).
I say this all as someone who likes and is a customer of DO. I am disappointed.
There are fundamental tradeoffs that happen when you take the VPS / cloud hosting route, and security is definitely one of them. There are reasons why Amazon, for example, doesn't just casually mix in their own services into AWS instances. Security is still a hard problem.