I lost all my data stored in DigitalOcean Spaces | Hacker News Reader

I lost all my data stored in DigitalOcean Spaces

medium.com

62 pointsnodegin30 comments

They were chasing me around last week with Google Adwords for a $100 free trial so I thought why not. Set up a droplet easy enough and invested 2-3 days part work in a test project for fun (didn't bother with external backups..).

Then without even bothering sending me an email they disabled my account to the likes of "We need to investigate that you are a real person, please send in docs A, B etc.". Normally you do that at the signup stage and when you approve someone for a trial you honour it. Now I cant even access my files.

Now been waiting 5 days to hear back from their security team after giving them the required docs. The regular support tells me the security team are "very busy at the moment". I guess someone decided to start offering $100 trials without thinking through the effect on the support team. Or maybe the campaign was too successful and this is their way of cost control?

If this is how they treat a big portion of their new trial users customers I doubt this will be a very long-term profitable campaign. I know that I will stick with my Google Cloud account for now.

Hi guys, thanks for your comments. Actually I do have backup in elsewhere so my website is still able to run, but surely this incident taught me never use DO services again.

Glad to hear it, I've been a big DO cheerleader and this is definitely making me re-think things, thanks for the write-up.

What I take greater issue with is that the author had to explain how it was impossible for him to accidentally delete the bucket (the files in it had to be deleted first) before DO admitted they had an issue.

I had a similar experience with moving my Ubuntu 16.04 image over to Debian. My Debian reimaging, where it warns me all data will be destroyed for an untouched Debian installation, had it's repos configured to use Ubuntu's, on my Debian reimage, which of course failed. At first they blamed me, suggesting I did something I can not remember, and then I changed the repos as they suggested and demanded an answer when it did not fix it. They admitted their repos have an issue that I can not solve and I have to destroy the machine completely to be able update or install to my Debian machine and lose my IP address that my DNS is set to.

They said this issue only happens when you reimage from Ubuntu 16.04 to Debian. Which I believed at the time since I was tired from this. I do not know if I believe them after the above post.

I remember having this same issue trying to do the same thing! Although my setup is different, the base problem was the same. Tried to reimage from Ubuntu to Debian only to realize their hypervisors are not setup to handle this as it came with all the same issues you mentioned.

Lesson learned? Backups are important.

Unreliability like this should be an expected irritation, not a disaster.

Both can be true. He should have had backups, but the service from DO was still very bad.

They key mistake here was lack of backups and backup procedures.

I wouldn't use DO Spaces or AWS S3 or others without backup, which means it's at minimum a separated account with separated credentials if it must be on the same provider otherwise good procedures to create the backup are sufficient.

Oh and sure, DO and AWS will promise you high and holy that the data will never be lost (99.99999999999999% permanence or whatever they write down). But no promise can prevent the eventual disaster, be it a software error, error by the provider or a meteor strike. At some point you will want your backups to restore.

I concur, the author of the post doesn't even talk about their own backups but assumes DO does it for them. I personally love DO but I'm not about to skip over my own security & resiliency measures.

Whether OP has backups or not, this is a very poor response from DO - they screwed up, not OP, and the compensation they've offered is nowhere near commensurate with the size of the screw up. Deleting customers data in an unrecoverable way is breaking the number 1 rule of hosting - it's not like a drive went out, DO specifically sell Spaces as a Backup solution!!

Wow, that seems bad. Does Digital Ocean provide any guarantees about data reliability? A 2 month refund seems way too low, maybe a refund of all money paid in the past related to storage would be closer to fair.

DigitalOcean spaces certainly has a few bugs and glitches that I have noticed while playing around with it. I don't have the confidence to do any production work with it yet. But it is also a relatively new product even though no excuse for losing data like that.

Btw, offsite backups folks, offsite backups. Always run offsite backups on any system

Since this post brought up the issue of backups, what do people use these days? I don't admin any servers at the moment, but when I did have a few side projects running I was a fan of HashBackup.

borg. proper open source unlike HashBackup.

restic has been very reliable in my experience.

Tarsnap: https://www.tarsnap.com/ It's a bit expensive for TONS of data, so it's not a great solution for your large video collection, Amazon Glacier or something of that sort for the larger data.

But for critical data, it can't be beat.

I use MySQLdump, tar, and sftp to store the files I need on another server. I know everyone's needs are different, but I would avoid any type of 3rd party/complex backup system. If you have to install some tool to restore your backup, you run the possibility of that tool becoming unavailable, or incompatible with your backups. Sticking to tools that have been around for a long time, and have multiple implementations mitigates that as a concern.

To see data being lost in the cloud is truly terrifying. I don't have the insight into DO or cloud management in general to assess whether this is a one-off. My guess is that most providers would go to great lengths to keep incidents such as these under the radar. DO didn't seem to be willing to go the extra mile (buck) and will no doubt pay a much bigger price now. Would be interesting to hear if the HN community has any insights into similar cases, regardless of provider.

I have been saying this for quite some time. Apart from two 'targeted' secruity incidents, Linode has been better at bandwidth, CPU perf, Storage Speed, and has had most of the features such as bandwidth pooling, private network for much longer then DO.

I don't understand why mainstream has all moved to DO instead.

They are also not VC-funded unlike DO, which makes them much less likely to follow the fate of GitHub.

It's called marketing.

Two is one and one is none. Drives fail, servers fail, keys get lost, accounts get compromised, planned deletions get fat-fingered.

Modern cloud APIs make it trivial to build redundancy into your system. Usage-based pricing and nearline storage makes redundancy relatively cheap in most cases.

It sucks that the DO data was lost; it's an awful situation to be in and should never happen given the way that cloud options are sold.

The moral of the story being that you should always diversify your cloud options and keep regular verified backups.

Nothing about this surprises me. DO rebooted one of my VPSes and it lost all network connectivity. I can't get any files off of it and can only access it from the console. Its frustrating as hell.

No, you lost all your data because you didn't make any backups.

Do you backup your S3 buckets? Digital Ocean Spaces is equivalent of that. So one would hope that would be stable enough to treat as a backup. Of course you have too important files, you can duplicate them, we backup our S3 buckets to another region, but another provider would be an option as well if we relied too much on them.

> So one would hope that would be stable enough to treat as a backup

Ah, ‘Hope’ - the most used, and least effective, backup strategy in the world.

You need to use paid prayers for better efficiency of this backup strategy. J

> Do you backup your S3 buckets?

Of course not, I'm not stupid.

Next excuse.

"What are backups?"

Reply on news.ycombinator.com