Hacker News Reader

Comment by xSwag | Hacker News Reader

TL;DR: In the DigitalOcean web panel you can check the "scrub data" checkbox when destroying a VM. When using the API this option is not ticked. This can lead to other customers being able to retrieve your data.

The author thinks that this is a security issue because this option should be enabled by default. However, (I assume) it's not in Digital Oceans interest to do full disk scrub because it reduces the lifespan of their SSD.

If a user forgets to log out of Facebook on a public computer, is it Facebook's responsibility? Similarly, if a user does not correctly delete data on a budget host, is it the hosts fault?

View on HN

The pro-DO bias of HN is showing again. If this was Amazaon or Linode, there would be endless hand-wringing and calls for lawsuits. When its DO doing something incredibly stupid and short-sighted, its 'blame the customer.' This is penny pinching at the cost of security, which is something we often complain about when it comes to other companies. Why does DO get a free pass?

"Oh, there's an option for that" to stop default dangerous behavior should not be excusable. This is further proof on why DO isn't enterprise ready and still a toy for barely stable dotcoms.

Thank god for EU privacy laws. If these guys have any EU presence, they'll be forced to clean up their act via regulation. Clearly, the invisible hand of the market and "there's a option for that" is the fail its always been in regards to security.

They have a datacenter in Amsterdam

Glad somebody pointed this out. It's also hilarious to me that we were all screaming for heads with the GD (mt) acquisition a few months back, and this is just a "pat on the back" type of thing.

Their growth and offerings are great, but they were going to run into problems sooner or later. As we've seen with many other hosts in the past, this is probably just the beginning.

I was with you until you started talking about the "invisible hand of the free market" failing and how instead we should be using State force to solve these problems. Isn't your comment a free market action, raising awareness about this problem? Won't this thread cause DO to take action? Say it doesn't. There are plenty of other hosting companies to switch to. And also there are times when I might be willing to trade security for convenience, price, practicality, etc. As a consumer, I want this choice. I think the free market is working fine. You can't bully your way to good security.

Wait a second. Why does this option exist at all? Are you saying there are people who DO NOT WANT their data destroyed when they destroy a VM? Who are these people?

It's probably there for DO's sake. SSDs have limited rewrite cycles, and each secure wipe will definitely shorten the lifespan of the drive much more than a simple quick wipe. This translates into cost savings.

Worst cost-saving measure ever.

Yes. This is a shitty default peddled as a benefit for the customer, when in fact it benefits DO.

No other provider charges you extra to erase your data when tearing down an instance.

It's called a dark pattern.

Most other provider use hard drives and they're different from SSD.

Okay, the misinformation is getting a little out of hand here. DigitalOcean certainly aren't charging you to erase your data, all you have to do is check the "Scrub Data" checkbox. Yes, you have to pay for the time your VPS is on while erasing, but that's perfectly reasonable.

Why would that be perfectly reasonable? They have an obligation to not make my data available to other people, I don't give a rats ass about the technical details of doing that.

"when in fact it benefits DO"

I could argue that it benefits the customer as well.

If it costs DO more (because of disk wearout) then in theory they would have to raise prices in order pay for their higher costs. And those higher costs could then disrupt the balance in pricing that makes them attractive.

My point is that forgetting the numbers you can't say it only benefits them. Being able to offer lower prices (because your expenses are lower) also benefits the customer as well.

If they can't afford to use SSDs properly they should not use SSDs.

It's much faster and there are certainly people who don't care because the data is worthless to anyone but them.

in other words, the only people who are affected are those who don't know the importance of scrubbing their data?

Come on people! DO is a developers playground ($5/month!!!!) not a place to host sensitive production data.

Sensitive is a relative term. You do development, you install a private key on the VM, you grant that key access to your system in order to test new integration prototype, you delete the VM before you disable the key - and voila, you got private key with access to your systems in a hands of a random stranger. Of course, the solution would be to disable the key access first - but how many devs are trained in security procedures to this level? How many can follow them flawlessly over repeated tedious cycles of development and resist the temptation of make shortcuts?

Proper procedures should make it easy for people to do security, otherwise they are part of the problem.

"Proper procedures should make it easy for people to do security, otherwise they are part of the problem."

Correct. And leaving the proper procedure in other people's hands is the problem. The best procedure [in testing a new intergration prototype] would be to create a new special test key and properly track and maintain that it is only enabled during the known testing time frame and then properly "disposing"/"revoking" it in the system YOU control. We can argue all day that DO should scrub the data by default and prevent new instances from low level access... however security relies on trust. So, do you TRUST (and KNOW) that checking the "scrub data" check box actually works and does what it says it does? If it doesn't, its beyond your control. And if you care about proper procedures and security you either a) don't use systems/servers beyond your control (if you are that concerned...) and/or b) rely on proper procedures and things you can and do control. I don't know what happens in DOs infrastructure... maybe there is some other leak I'm unaware of so to limit any damage for testing I will properly create and control short lived test keys.

I use DO for bootstrapping and trying out new ideas, programs & co. There is no information I can think about I would wish to get wiped off of the SSD.

Source code? SSL private keys? Private keys used for deploying code or configs from Github? New Relic license codes? AWS access keys? Pcap dumps? Database passwords? Deleted blog posts? irssi cybersex paged out to a swapfile?

Seriously, if there's nothing on your cheap sandbox server that you don't want published, you're probably not using it in the first place.

Reading your comments in the link, and comments in this thread, I find you to be alarmist at best.

Yes, this issue needs to be raised. No, you can't say that anyone who doesn't agree is a non-factor.

You have the option to scrub your data anytime you like, I've always understood that wasn't the default on DO.

Then certainly you don't mind giving me your root login?

To a server you have discarded?

If you care about your data then the erase will not protect you enough anyway, if you think it does then you've probably misunderstood the underlying architecture a little.

The data for your virtual machine's virtual disk(s) will not always be in the same place, it may get moved around the storage volume or moved between storage volumes and this will be completely transparent to you. It may also get stored on backup volumes too. When you "secure wipe" the disk as you kill the VM all you are doing it wiping the data that is in the current location not any latent copies that may be sat elsewhere (as backups or "ghost" data sat in currently unallocated bits of physical media).

The only way to grantee your data is stored securely and is gone securely when you want it gone, is to use full-volume encryption from the start and make sure that the key's are never stored at the provider side (this does mean that you need a mechanism by which you can provide the key(s) to the VM whenever it reboots for some reason). That way there is no need to wipe the current store or worry about ghost data elsewhere: just destroy all copies of the keys for those volumes. Of course there is still some risk as the keys need to be in RAM somewhere so the encrypted volumes can be accessed at all, but once you get to that level of concern you can only be sure you are secure by having your own physical kit.

Of course not all current hardware sharing solutions can support full volume encryption as you don't really have a proper volume(s) that you can encrypt and put filesystems in, just a part of a larger filesystem pretending to be one (and you can't mount things so you can't use a file-based volume instead)...

tl;dr: wiping the VM's disks does NOT protect you from this sort of thing, using properly implemented full volume encryption will (as much as is possible in a shared virtualisation host, iff your hosts's solution can support FVE in the guest VMs/containers at all).

You make it sound like there's no valid point of security in between "anti-the-next-random-guy" and "anti-sophisticated-governments." There are a lot of people who care about their data enough that it shouldn't be left lying around for passersby, but not so much that they're going to implement FDE. I'd call them "normal people."

The suggestion of using TRIM does not (in all cases) protect against accidental data visibility to the "next random guy", it doesn't even protect from some other current random guy as your container could have moved and left another user with access as their new container got created over the space you used to occupy - so unless you have another method that hasn't been discussed yet there isn't a point in between: you protect yourself from both or you protect yourself from neither.

If you need to be sure your instructions to wipe data do protect you from that random other guy then you either need to use good encryption (it doesn't have to be full volume encryption but once you get to the point of caring it is probably easier to go the whole hog than do it piecemeal) or you need to have dedicated physical storage (something a VPS provider generally doesn't offer.

Of course the provider could instigate full data wiping for all relevant operations, but that imposes I/O load that will affect all other users of the given host machine(s) unnecessarily (I say "unnecessarily" because most won't care as their bulk data is not sensitive and they've invalidated associated things like SSH keys anyway, the host will take the attitude that if your data is sensitive you need to take measures to protect it).

If it was a lifespan issue they could just encrypt the partitions (minimal overhead with AES) and then destroy the key with the droplet.

Due to the way that SSDs wear level, even if every block returns 0, you don't know if that's the same block or if the real data is squirrelled away on out-of-life blocks. From what I understand the SSD has a lot more capacity than advertised and moves the blocks forward as sections of the chip wear out. Assuming you could reprogram the controller (or use a different one), you could go back and read the old blocks and recover data in the clear.

Not several times more capacity, it's generally about 7% (that's why you often see 120 GB or 240 GB drives that are in reality 128 GB or 256 GB.)

You're right. Encryption is the right answer. What does Amazon do with their SSDs? Anyone know?

I would expect any sane large provider to zero the disks simply to make it easier to overcommit on the storage layer.

The problem with encryption at that level is that trim no longer works as there are no longer any zeroed blocks - so writes are slower. Also there will be more write applification as every write will cause a block to be erased.

Of course it works. The filesystem knows which blocks the underlying storage device can forget and tell it using TRIM. That the content of the block was encrypted is irrelevant. TRIM is for putting blocks in some undefined state when no longer needed, not for zeroing them.

See http://worldsmostsecret.blogspot.com/2012/04/how-to-activate...

Oh it looks like I'm a bit out of date, it took years but I can see they now support this. Of course this comes at the cost of being less secure (giving away the used block locations), though it shouldn't matter for this use case.

I'm surprised I didn't see anyone raise the issue of Data Protection given DO's European presence. This comment below [1] appears to confirm that customer data is mishandled in violation of EU Data Protection law. When you ask 'is it the hosts fault?' I think the answer is a most definite YES.

Surely some lawyer out there took a look at this, so maybe I have missed something, but this looks like a big problem to me.

From my own experience using DO I can say I'm a happy customer and I plan to keep using them. I tick that box when I've used anything remotely sensitive in a VM when destroying it, and leave it empty other times (like when I've made a mess experimenting with something and want to quickly trash & re-create a droplet).

[1] https://news.ycombinator.com/item?id=6983260

DO is a data processor under EU data protection law, while the customer would be the data controller. EU data protection law currently (it will change with the new regulations) only imposes legal duties on the data controller. As such, it is the customer's legal problem if it (or its data processor) has failed to handle personal data correctly.

> Similarly, if a user does not correctly delete data on a budget host, is it the hosts fault?

What does the verb "destroy" mean?

Screenshot taken today (30DEC13) from a fresh blank Digital Ocean server I just provisioned:

http://i.imgur.com/fJOxRN9.png

What are we looking at in your screenshot? I've been Where's-Waldoing it for the word 'destroy' but can't find it.

That's someone else's data (three someone elses' if you count the iPhone end users) read from the root block device on a minutes-old brand new fresh Digital Ocean VM that I got from them for a $5 PayPal payment. It had been mkfs'd but not zeroed.

Command was:

    apt-get -y install binutils ; dd if=/dev/vda bs=1M | strings -n 100 | grep 2013-12

The destroy api call docs are here:

https://developers.digitalocean.com

(It's the /droplets/[droplet_id]/destroy one.)

I think he's showing that the data is not "destroyed" in the sense it still exists. Destroy in a virt context doesn't necessarily mean "destroy all the resources associated with a VM"--I don't know about DO's product offering, but at the hypervisor level, at least with Xen and libvirt, you often want to "destroy" the instance (forcibly terminate/undefine from the hypervisor) and leave the resources (storage pools/devices, IP pools/addresses, network flows/filters etc). I think focusing on the word "destroy" is a bit of a canard; the real problem is insecure defaults wrt block device scrubbing when you issue an API "destroy" (which wouldn't be any better if it was called "delete" or "undefine").

nice post - also #fmc says hi ;)

The term "disc scrub" makes little to no sense in terms of SSDs, which makes this issue even more complicated. People who work in the disc recovery field have been dealing with this since SSDs became popular.

An SSD is limited by its number of writes. To compensate for this, the SSD has very complicated on board logic that abstracts the actual SSD away from what it tells the OS system. This allows it to do certain tricks to save writes. However, when you are "scrubbing" an SSD, internally the SSD might be writing somewhere else entirely. Scrubbing is not considered an effective way of wiping SSDs, from what I believe.

Ok, so the data isn't necessarily destroyed immediately after a scrub. But how does that play out at the level of VM users? Is there a normal usage scenario where the portion of the SSD containing my deleted data is made readable to someone else's VM, or will it be inaccessible to normal VM users until it's overwritten?

I'm not an expert on this, but I believe that at the VM user level they would see wiped data because of the internal mapping. I think physical analysis of the drive would be required.

There is a vast difference between writing out zeroes to the SSD but still having some of the original data potentially persist on the SSD but unreachable without special techniques, and not zeroing out the SSD and giving the device to a new VM and letting it trivially access everything that was previously there.

If I can provision a new VM and cat /dev/vda and see data from the VM that previously occupied that spot, then you are doing it horribly, horribly, horribly wrong.

That zeroing out the data leaves open a different and vastly more difficult attack path doesn't make that any less true.

They don't have to do a full disk scrub on the physical SSD to fix this security problem. All they have to do is what every sane sparse disk image implementation in the history of mankind has done: catch reads from unwritten blocks in software and return a block of all zeros.

I think it's less in Digital Ocean's interest to silently give customer data to a subsequent user of that block device?

Why?

If customers don't care enough about the data why should DO?

DO is a budget provider. Their main hook was cheap SSDs, and is still cheap costs. They billable time it takes to wipe a VM adds to the cost. If you want the data zeroed... then pay for it, otherwise it must not be important enough.

DO is essentially saying they're using insecure defaults by design, which is a Really Bad Idea, even if it prima facie saves time/money. It's relatively easy for them to post a point in a FAQ about why you're billed for x minutes after an instance is destroyed, and include the request params/headers to avoid it. On the other hand, it's effectively impossible to unleak an inadvertently leaked apple developer signing key because an unwitting customer didn't rtfm.

Reply on news.ycombinator.com