Hacker News Reader

We believe the data to be extremely DURABLE (survive), but if your strategy is to "host content" in a highly available fashion where people will die if your content is offline for an hour, we recommend you use two different providers with some sort of fail over

Do your durability metrics take datacenter failure or human error into account?

Datacenter failures are rare, but they do happen and can cause data loss if all of your data is in that data center.

Likewise, human error can cause cascading failures across a datacenter (or beyond) if there are no firewalls between zones that prevent a single person/command/software update from affecting all copies of the data.

That depends on what you mean by failure. Are you talking about a data center failing because every single machine inside blew up? Otherwise the common failures like DC power outage or network drops are about availability rather than durability. The data is stills safe on multiple drives.

I'm talking about the kind of failure that hit a Microsoft Azure data center:

https://www.datacenterknowledge.com/microsoft/azure-outage-p...

“ but in this instance, temperatures increased so quickly in parts of the data center that some hardware was damaged before it could shut down" ... "A significant number of storage servers were damaged, as well as a small number of network devices and power units.”

Yea if that damaged all the storage servers containing your data then there would be data loss.

That's why I asked if that 99.999999999% durability number includes datacenter loss. It's an unlikely failure mode but is it .00000000001% unlikely? I don't know.

Given the fact that Azure lost a datacenter with this failure mode, I don't think it's in the "likelihood of an asteroid destroying Earth within a million years" ballpark.

Their durability page doesn't really clear it up, they say "Because at these probability levels, it’s far more likely that ... Earthquakes / floods / pests / or other events" known as “Acts of God” destroy multiple data centers". But from the post above: "any one file (or piece of file) is in exactly one datacenter."

So it doesn't take multiple datacenter failures to lose data, just one unless you explicitly copy your data to multiple datacenters.

If a lightning strike is close enough the surge protector won’t save your equipment. When I read the overview of this I expected a roomful of fried servers, not an orderly shutdown triggered by a cooling failure.

To be a little more specific, the "2" in 3-2-1 is for two different media types. Hard drive + tape, for example. [Edit: ok it doesn't have to be "types". But two different media - don't put all your backups on one disk!]

Yev from Backblaze here -> Or Hard Drive (Internal) + Hard Drive (External) - that's what we typically see!

Last I checked, the backup service exclusively used the Sacramento DC. Has this changed? Being in the Sacramento area myself, I'd be much more comfortable if my offsite backup was more than a couple miles away.

No disaster recovery? :/

Same as every other storage provider's default/basic storage offering. If you want georedundancy, you will need to build it.

EDIT: Apparently GCS has this feature built in. Did not know, very cool!

I thought basic products like S3 provided cross-region replication, which gives georedundancy?

But anyway why should I have to build it on top of the provider's offering - why wouldn't they provide georedundancy for me? It seems like a truly basic thing to expect for a backup solution?

But I'm not an expert in this area.

Good Cloud Storage (GCS) has this functionality out of the box.

https://cloud.google.com/storage/docs/locations

/I work for Google/

S3 replicates across available zones, meaning copies in multiple DCs but in the same general area.

You can setup a bucket to replicate a bucket in another region, at double the storage costs plus bandwidth charges.

All clouds have options to do multi-regional storage. GCS has multiregional class. Azure has GRS class. AWS has cross-region replication that can be added to a bucket.