Hacker News Reader

Comment by jacquesm | Hacker News Reader

Interestingly: S3 loses based on their own figures some data every year. They store 100 trillion objects and they say they are 99.999999999% reliable on an annual basis. So every year they lose some objects, but the chance that these are your objects is vanishingly small.

View on HN

Nitpick: they actually make a distinction between availability and durability. Availability means "you can get your data." Durability means "we still have your data." Only durability has 11 9's.

Right, but we are talking about data loss, so I think we are all talking about durability.

Yes, I should have used durability. Thank you for the correction.

Be careful with projecting these numbers forward. As far as I can tell, there’s no SLA for durability, only for 99.9% uptime (you can get 10% off your bill if this is violated)

> Be careful with projecting these numbers forward. As far as I can tell, there’s no SLA for durability, only for 99.9% uptime (you can get 10% off your bill if this is violated)

> Designed to provide 99.999999999% durability and 99.99% availability of objects over a given year.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDu...

This isn’t in their SLA but they do post it.

Ahh interesting, I thought this was a calculated reliability rate, not an actualized one.

You'd hope that with such a massive statistical base that the two would approach each other?

Well if they’ve never lost an object then how would you go about doing that.

Statistics can do that. "If we were only X% durable, what are the odds we would have made it so far with no data loss?"

Then pick a threshold p value you like and that X% will be a reasonably confident estimate

If other words if you think you have 99.999999999 but you really have 99.9, Nature will correct you rather quickly. If reality persistently lets you get away with it, you can counterfactually deduce you must have more nines.

You’d have to randomly sample the set of all objects to do that. But I’m not aware of a database technique that lets you pick randomly. So I’m not sure how you’d validate the objects exist in practice. Checking 100 trillion objects seems expensive and time consuming. Doing it in flight while objects are being accessed is free but also heavily biased (ie those are files you’d be more likely to repair in the first place anyway)

I don’t understand how to calculate those odds though. Like is it a continuous rate of event probability?

There's different sources of failures, and when you start claiming 11 nines of durability, you really have to go and think about all the different possible causes of failure. So it's not just a flat continuous rate, it's a sum of many different things.

11 nines is such a big number, you start modelling natural catastrophes, floods, solar flares, very unlikely discrete events as well.

With those figures, that would be 1000 objects per year.

Reply on news.ycombinator.com