Hacker News Reader

Comment by mdekkers | Hacker News Reader

Could you provide more details?

On delete of VMs, the next user to be allocated disk would be able to read your data. There was a "scrub" feature that would bill you longer/extra to zeroize your data and not give it to the next customer. Apparently they hadn't heard of TRIM or thin provisioning.

https://github.com/fog/fog/issues/2525

https://news.ycombinator.com/item?id=6983097

https://news.ycombinator.com/item?id=6983520

Some of the comments by DO on that GitHub thread, are really surprising, and disappointing to say the least...

I wrote those comments, fog wasn't using the API as he would have liked, and then this guy starts a bunch of trouble on HN stating that there was a vun on DigitalOcean, and conflated someone using the API incorrectly with us having a security vun. Two things I was trying to (admittedly not very well) say was: if DO did have a security issue, posting it on HN ("DigitalOcean leaks customer data between VMs" huh?) instead of emailing us about it isn't a very responsible way to disclose it and, there isn't actually a security issue, the fog app wasn't passing the scrub flag. In retrospect, I should have been less of a jerk, but I still think sneak was acting in bad faith in how he went about the whole thing. 10 years later and I still remember how pissed off I was that day, hah.

Wow, this just confirms my belief.

There is precisely zero circumstance where it is ok to give private customer data to another customer.

The fact that you think it had anything to do with an API or how it is used is all anyone needs to know about it.

The idea that warning your customers of your vulnerabilities is "irresponsible" is only true if you care more about revenues than your customers' security.

Unless I misunderstood something and one has to explicitly opt out of scrub for this to occur, I agree 100%. This is surprising. And they were mad that a customer thinks this is undesirable?

I was using DO. It is not a serious project but I think I'll be moving on nonetheless.

For what it's worth, neom has moved on from DO a while back.

Yeah, I read a bit further and it was not scrub by default but had to be opt-in. And today is the first I heard that all API keys have ridiculous access. I'm done.

Now I just need to find a cloud hosting service that will do a low cost easy k8s cluster, preferably not AWS.

Looks like OVH does managed k8s, and they're pretty solid at this point for the most part. https://us.ovhcloud.com/public-cloud/kubernetes/

Thank you, I didn't know they did. That puts them top-of-list for me.

> there isn't actually a security issue, the fog app wasn't passing the scrub flag

At a certain point a default behaviour can be so bad, and so clearly not what a user would expect or want, that it constitutes a security issue. I would think this _more_ than qualifies.

You're right. I hadn't really thought about it that way. My response at the time actually came from confusion on my part (that is my own fault I suppose). I got woken up at 4am on a Sunday by a flood of calls saying there is a post on the top of HN about DO leaking data between VMs. I got out of bed to see if I needed to wake up the engineering team, checked the support system and the disclosure emails because my first thought was shit someone has pwned the LXC/escaped the jails or something. Then when I read the github, I thought the scrub flag wasn't being respected when it was passed. That was the type of security issue I had originally taken it to mean, both I would have woken up engineering for. It was a someone is in the hypervisor type of security issue I thought we were dealing with. Once I realized it was fog didn't pass the scrub flag, I wish I'd just gone back to bed and dealt with it on Monday instead of trying to do it in real time, that was stupid of me. I also accept your point that an unexpected behaviour can in itself be a security issue.

> there isn't actually a security issue

The official blog from DigitalOcean is here:

https://www.digitalocean.com/blog/transparency-regarding-dat...

While it does not call it a “security issue” directly, it details several changes to the service to prevent customer data leaking between accounts. That seems to contradict your position that there was no problem.

> 10 years later and I still remember how pissed off I was that day, hah.

I would have hoped that after 10 years you would be able to admit that letting customers read each other’s data was a mistake. The existence of a disk scrub API, the problem being improperly reported, and DO being advertised as “not for production use” are not valid excuses.

The fact that you think it should be the customers’ responsibility to make sure that DO doesn’t accidentally give your private data away to unknown third parties is an absolutely minblowing take on this matter.

I’ve been a DO customer for the better part of a decade, but there is no way that I can continue to be a customer with a company that has such a blasé stance toward security and protecting customer data.

If anyone can suggest any alternatives that are not Hetzner, I would be interested.

Well first, I've not worked for DigitalOcean in a long time, I left in 2015. I don't really wanna re-hash something from 2013, but again: when you deleted a VM you could click the scrub button or pass the scrub flag in the API when you issued a delete call. It was very well documented and the scrub button wasn't hidden anywhere in the interface. The fog platform wasn't using the scrub flag in their app for whatever reason. As I recall, we did change it to scrub by default shortly after. From what I remember, the reason it wasn't originally default was the zero'ing process tried up a lot of resource on the box and it was hard on the drives and we were very much in startup mode at that time with limited capacity, so it was better to only zero if the customer requests it.

You still don't appear to get it though. The fact that there even exists an option (never mind a default) which does not scrub storage between use by different tenants is completely unacceptable in a multi-tenant environment.

> tried (sic) up a lot of resources on the box

Do you know what else ties up a lot of resources? Civil action, negligence claims, PR disasters.

This specific GitHub issue is one of several why I will never use DO: language, official response, and attitude all matter. I'd be curious to know how much revenue DO has lost because of this one issue.

Imagine if your bank told you they only close the vault when a customer asks.

And they give you an excuse that it’s time consuming for them and you should respect their time.

Would you bank with them?

This is a pathetic response and you are dragging your former company through the mud by putting the incompetence that you participated in, front and center.

It’s amazing that this happened so long ago and yet you seemed to have learned nothing from it, and now want to double down on your right to leak sensitive data (because it was convenient at the time)

I was always very clear at that time "this is a startup" - "do not use DO for mission critical work" - "the product is not very mature yet" - we had no sales team, nothing in our marketing was gear towards business and we primarily catered to students/personal sites/etc. I'd accept your analogy except we regularly said "use AWS for important stuff, not us".

Even startups have to follow the law.

Before the GDPR the EU had the 1995 Data Protection Directive, which you would have to comply with to have EU customers.

You can’t just decide to not follow the law because you’re a startup.

What law would cover this??

I just told you in my previous comment..

In the event your google is broken:

----

https://en.wikipedia.org/wiki/Data_Protection_Directive

>the controller must implement appropriate technical and organizational measures to protect personal data against accidental or unlawful destruction or accidental loss, alteration, unauthorized disclosure or access, in particular where the processing involves the transmission of data over a network, and against all other unlawful forms of processing.

https://en.wikipedia.org/wiki/Personal_data

Are you confused about what PII is?

Should I explain this too?

I do find the claim that "there's nothing irresponsible about full immediate disclosure" to be interesting.

https://github.com/fog/fog/issues/2525#issuecomment-31336855

Consider the multiple S3 leaks that caused AWS to change their defaults to block public access and disable ACLs. You could say that since buckets were always default private, any misconfiguration was simply user error and users complaining were doing so in bad faith. Or you could understand the implications and help your customers protect their data.

Why on Earth would someone have to opt-in to scrubbing with a flag vs. opting out? That's surprising. Is the flag on by default and they specifically opted out before complaining about it?

The scrub flag was default false, apparently because scrubbing was slow and caused wear.