This seems like a non-issue to me. If you're using an IaaS provider you should be treating the network as volatile from the get-go. This is the reason AWS has things like auto-scaling groups. You should be designing for failure in "the cloud"
These facts combined certainly make it easy for the younger generation to forget that redundancy and backup still don't replace each other (never did).
(Could be worse. Their first lessons could be like mine, in C! If anything will make you paranoid, writing C will.)
At one of my positions we had stupidly put too many eggs in the basket of a single physical machine. Its disk controller failed in a way that it trashed the data volumes. I was unable to convince anyone that "move to Amazon" was not a one-step solution to "how do we make sure this never happens again".
To be fair the point of the cloud is that they deal with redundancy, HA, distributing to multiple datacenters, etc. through their services - but you need to use and understand implications of said services to leverage that.
But that doesn't stop the misconception.
The lack of explanation is what worries me most - it leads me to think this might have been a case of "we forgot to replace a bad drive, then the second in the pair failed".
Fire happens, floods happen, electrical faults happen, mistakes happen. I can't really expect any human or humans to be perfect.
And many years ago it suffered a truck crash ( the truck crashed on the lamp post right outside the server company building and destroyed the telecom-related wires)
DO should be designing for failure: VM storage should be on a SAN. A single physical server failing should not cause data loss. This is basic stuff.
Does any cloud provider have SAN backed VMs? If any do, at what price?
SAN reliability ratings start at 99.999%.
I actually prefer the lower-level abstraction: if you want a lower failure rate (or higher speed), you can RAID together attached EBS volumes yourself on the client side and work with the resultant logical volume.
For ephemeral business-tier nodes, EBS gives you a few advantages, but none of them are that astounding:
• the ability to "scale hot" by "pausing" (i.e. powering off) the instances you aren't using rather than terminating them, then un-pausing them when you need them again;
• the ability for EC2 to move your instances between VM hosts when Xen maintenance needs to be done, rather than forcibly terminating them. (Which only really matters if you've got circuit-switched connections without auto-reconnect—the same kind of systems where you'd be forced into doing e.g. Erlang hot-upgrades.)
• the ability to RAID0 EBS volumes together to get more IOPS, unlike instance storage. (But that isn't an inherent property of EBS being network-attached; it's just a property of EBS providing bus bandwidth that scales with the number of volumes attached, where the instance storage is just regular logical volumes that all probably sit on the same local VM host disk. A different host could get the same effect by allocating users isolated local physical disks per instance, such that attaching two volumes gives you two real PVs to RAID.)
• the ability to quickly attach and detach volumes containing large datasets, allowing you to zero-copy "pass" a data set between instances. Anything that can be done with Docker "data volumes" can be done with EBS volumes too. You can create a processing pipeline where each stage is represented as a pre-made AMI, where each VM is spawned in turn with the same "working state" EBS volume attached; modifies it; and then terminates. Alternately, you can have an EC2 instance that attaches, modifies, and detaches a thousand EBS volumes in turn. (I think this is how Amazon expected people would use AWS originally—the AMI+EBS abstractions, as designed, are extremely amenable to being used in the way most people use Docker images and data-volumes. The "AMI marketplace" makes perfect sense when you imagine Docker images in place of AMIs, too. Amazon just didn't consider that the cost for running complete OS VMs, and storing complete OS boot volumes, might be too high to facilitate that approach very well. Unikernels might bring this back, though.)
Seems like they sell 50 GB of real SAN backed storage for just 3.6€ per month.
How can they afford to be so cheap?
Disclaimer: happy customer
Except that is not SAN backed storage.
From the website:
> We only use RAID-60 drive arrays with a minimum of 16 drives
That's somewhat scary. RAID-6 might be almost ok, but striped? No thank you. I bet they don't also have block level checksums.
Agreed.
> the customer should read them carefully.
Sure, but they shouldn't need to, as Digital Ocean should set expectations clearly.
Keep in mind Digital Ocean's <title>: Simple Cloud Infrastructure. Being surprised because there's an unsafe default hidden in a document somewhere didn't work out for MongoDB and it won't work out for DO.
They bought a hugely expensive SAN solution (HP I think). One of my questions was: What happens when the SAN fails? Well you see, because it has redundancy, that can't happen. Clever uh?.
First time it failed, everything gridded to a halt for two days. The second time they did better and had it running within eight hours.
That's completely different from suggesting 'SANs are magical boxes incapable of losing data'. Have you considered apologising?
But if DO had a multi-SAN setup with synchronous replication this thread wouldn't exist because DO's business model of "super cheap VPS w/ fast SSD storage" would have failed due to costs. Everybody wants Five Nines until they have to pay for it...
The implicit (and explicit) volatility of cloud hosting should change that expectation with such services.
"Non-issue" is an exaggeration because it was potentially a catastrophe. But this is sort of the way these services "work." Servers/droplets/instances are ephemeral and replaceable, and their underlying data is not guaranteed in a failure.