Best Twitter comment (grammar errors and all): "And if your full business relies on one tech partner (no offsite backups) your not doing your tech job right."
- No offsite backups? Agreed. Even for a two person team it is sloppy.
- "Relies on one tech partner?" Strongly disagree. Even large enterprises often have a hard dependency on AWS, Azure, Rackspace, or similar. To suggest that a two person team should have deployment plans for multiple independent cloud vendors is just fantastical thinking with no basis in reality.
Plus if they did over-engineer it by making it cloud agnostic, setting up accounts to sit dormant, cold instances elsewhere, etc, people would just criticize them for that inefficiency/wasting time.
How you got from "single tech partner" to "have no disaster recovery plan" I don't know.
Very, very few tech companies could simply move everything to a new cloud provider in a few hours. I would even hazard a guess that almost none can.
I have all my infrastructure as code and can break it all down and spin it back up in kubernetes clusters in minutes. But due to the quirks of each cloud provider, there are tons of little fixes that would inevitably need to be made.
Not to mention that many companies have way more data than could even be copied over in a few hours.
Yes, ideally they would have already tested their back-up solution, the back-ups would be offsite and, if something like this happened, they could stand-up on another provider. But that just ignores the reality of them being a super tiny business. Almost no one at that size is going to do that.
But I still support that DO here is a 100% liable toward their client. Now the liability between the said client and his own clients is an other matter.
What happened w/ Digital Ocean is inexcusable, and has potentially dire consequences for two individuals livlihood. In the immediate aftermath of such an event, focusing on the devs percieved lack of disaster preparedness seems petty.
I think we needlessly shame one-person operations for focusing on actual customer needs instead of ops busywork and yak shaving.
No one that I've seen so far is saying they should have a system that is "muilti-site, fault-tolerant, self-healing, webscale that Google would be proud to have".
They are saying run a simple backup and keep it literally anywhere else.
I think we needlessly hyperbolize "do a backup once a month and keep it somewhere else" into some sort of NSA operation.
Its backups. Its 2019. Its dead-easy and very affordable.
> Digital Ocean "Trust and Safety"
Does this phrase give anyone else the heebie-jeebies? They deliberately locked his account without looking at previous metrics over the previous months to conclude this was obviously not a concern, and why?
Things that have crossed my mind:
Is their automatic system was poorly tuned? Was this deliberately initiated? If so, why?
Ideally partners should be trusted (and trustful), in practice, they aren't
Though trusting DO/AWS/GCP, etc is much more reliable, than, let's say, betting your whole business on somebody's proprietary API (like an FB game, an Linkedin API something, etc)