Hacker News Reader

Comment by idlewords | Hacker News Reader

What is the meaning of the claim about "99.999999999% annual durability"? Does that mean one chance in 100B of an object being unretrievable?

View on HN

Backblaze wrote a good piece on that https://www.backblaze.com/blog/cloud-storage-durability/

Thank you for that link! It's very interesting to me that all the calculations laid out here assume independent failure events.

It is universal practice within cloud service providers to span redundancies across "fault domains" - basically things which could make failures correlated, like being in the same machine/power strip/datacenter/geo region. If you assume your fault domain analysis is good, then your failures should be independent. Many global outages are the result of a previously-unidentified fault domain, like the Azure certificate issues. Of course past a certain point it becomes unimportant - who cares about your data if an asteroid takes out every datacenter on Earth at once?

Nothing in engineering is 100%. As far as engineering goes 11 9s is pretty much as good as you can get. For comparison, AWS S3 and Glacier are 11 9s durability too.

It's worth bearing in mind the difference between durability and availability. Durability is roughly the chance of losing your data over a given time span (in this case a year), whereas availability is about how reliably you can access the data (and is almost certainly a lot lower than 11 9s). A service can be very durable but have very poor availability.

"What is the meaning of the claim about "99.999999999% annual durability"?"

It has no meaning whatsoever. Someone on the marketing side of the team decided that was a "competitive" number to present, outwards, and someone in engineering was tasked with, working backward from that number, coming up with some plausible calculation that resulted in it.

In the real world, they, like Azure and Amazon, will have single point in time outages that will wipe that out for a year or more.

Here is what an honest assessment looks like:[1]

"Historically (updated April, 2019) we have maintained 99.95% or better availability. It is typical for our storage arrays to have 100+ day uptimes and entire years have passed without interruption to particular offsite storage arrays."

...

"In the event of a conflict between data integrity and uptime, rsync.net will ALWAYS choose data integrity."

[1] https://www.rsync.net/resources/notices/sla.html

> In the real world, they, like Azure and Amazon, will have single point in time outages that will wipe that out for a year or more.

An outage affects availability, but as long as it's not permanent it doesn't affect durability. For example, if I add a new backup provider that stores data on-premise I've added a (nearly) independent data store. This substantially decreases my risk of losing my data unrecoverably (increases durability) but if I don't set up any sort of automatic failover I'm still at risk for substantial outages (no practical increase in availability).

For example, I don't believe Amazon has ever lost any S3 data (https://www.quora.com/Has-Amazon-S3-ever-lost-data-permanent...), and if they did it would be a big deal. Same with the other major cloud storage providers.

> Someone on the marketing side of the team decided that was a "competitive" number to present, outwards, and someone in engineering was tasked with, working backward from that number, coming up with some plausible calculation that resulted in it.

I would be incredibly surprised if that happened. That's not the way I've seen anyone work here.

(Disclosure: I work at Google, though not in Cloud)

You are mixing availability (access at any given moment) with durability (not losing data). From the FAQ:

Cloud Storage is designed for 99.999999999% (11 9's) annual durability, which is appropriate for even primary storage and business-critical applications. This high durability level is achieved through erasure coding that stores data pieces redundantly across multiple devices located in multiple availability zones.

Disclaimer: I work at GCP, although not in GCS specifically.

"You are mixing availability (access at any given moment) with durability (not losing data)."

You are correct - I misread that as availability even after quoting that very same line.

Reply on news.ycombinator.com