Hacker News Reader

Comment by jofe | Hacker News Reader

Hey folks, I'm the CSO over at DigitalOcean (verification: https://keybase.io/custos). I'm going to look into this so we can figure out what happened here. Garland, I've temporarily set up an email alias which is my HN username [@] digitalocean.com - can you please shoot me some info?

View on HN

Hi jofe. Without wanting you to comment on specific cases, as a customer paying you $1k per month stories like this terrify me, and it's not the first time I've seen something like this on HN. Can you definitively say whether this kind of thing has ever happened in the past and if it could happen again in the future?

Hey drcongo. Things like this occur at every cloud provider and I would never be comfortable promising you it's not a possibility. What I can tell you is that DO invests very heavily in trust and security (moreso than any company of our size that I've seen in my career). While we are far from perfect, we have a ton of controls in-place to ensure things like this don't happen, especially for customers with your type of track record.

Always happy to chat live with you, just open up a ticket or email me at the address above.

That's a fair response, and I totally take threatofrain's point below too. I guess a better question would have been: is there anything as a customer I can do to prevent this?

Yes, backup your data somewhere else.

(1) Gitlab/Github for source code (almost free)

(2) automysqlbackup (or similar) to an S3 bucket for your database (very cheap) using S3FS

(3) Make sure you have a way to reproduce your entire server environment using fabric, ansible, chef, puppet (or Docker images) that is also in source control

If you have PROD code, this is normally called DR (Disaster Recovery)

Hey drcongo. Jarland from DigitalOcean here. Truthfully, we would have to talk about individual cases to provide a detailed answer. Being aware of who you are and knowing how the details of these cases compare in relation to you, I would say that you already do everything that you should to prevent being caught up in the kind of experience that has you concerned.

I realize that is vague, but it's a bit of a difficult thing to discuss without exposing private data. If I so much as say "Just don't do X" then I'm effectively saying Client A did X. Tough waters to navigate.

I hope that helps a bit at least.

Perfect, thanks for taking the time to answer.

Hey drcongo, I'll tell you anything you want to know about my case either public or private.

At least in Tokyo I know plenty of people who are running PG on DO in a docker instances and just doing backups on DO cause its cheaper than something like heroku managed PG so if I was one of them (I kept my data layer off of DO) I would be completely destroyed by this. So I tell everyone I know now at meetups about this experience and so happy to share specific details.

They told me I log on from multiple locations (OMG I have a laptop and work while I travel, busted) and that I had a gmail address on my account.

The most agregious thing is that if their trigger happy "security and trust" flagger flags you, there is no warning where they reach out to you, they just destroy your instances, don't tell you about that either until you logon and are asked to file a ticket if you want your account unlocked...then a month later will unlock it for you but everything nuked.

I don't know about other providers, but I don't how the handling of any false positive from flagging is to not give any warning beforehand, destroy all instances, and not even give a courtesy mail after destroying all instances and locking accounts. I had to hear from a user that our site was down...(we were also in the middle of an expensive bio related computation that was 3 days in when all was destroyed)

Hi had heard a few stories like this here too but I figured it was people doing spamming or gambling or something like that. We are a biotech startup and also around $1k per month with our usage, and it was completely shocking.

Thanks for the message, I will send you an email with the details if you want to look into it. I'll gladly add any resolution etc to the write-up I am doing. Thanks.

Reply on news.ycombinator.com