One interpretation is that about 1 bit per 100 GB will randomly flip each year. That or S3 Glacier is expecting to hit a catastrophic event every 100 billion years (which doesn't seem nearly frequent enough).
Backblaze apparently gets the same number by looking at the failure rate of their drives, and how long it takes to recover from a drive failure (reaching the same redundancy as before the failure). By being able to recover from three drive failures before the first drive is rebuilt, they get 11 nines of durability [1]. Or in other words: 0.000000001% of drive failures are non-recoverable and lose data. I assume AWS does basically the same calculation, adjusted for whatever technology they use.
1: https://www.theregister.com/2018/07/19/data_durability_state...
https://blog.synology.com/data-durability
> Data durability: the ability to keep the stored data consistent, intact without the influence of bit rot, drive failures, or any form of corruption. 99.999999999% (11 nines) durability means that if you store 10 million objects, then you expect to lose an object of your data every 10,000 years.
Edit: and it'll still break when us-east-1 is destroyed by the sun exploding.
Shouldn't that be multiplied by the number of bytes of this object? I still don't quite understand the math apparently because then we'd all be losing 10KB+ files yearly.
For the purposes of this calculation, you would probably use the average size of a stored file, determine the number of chunks that was split into, and then the likelihood of losing/corrupting a chunk.
By looking at the durability of objects, you (a) reinforce the object-store nature of S3, and (b) avoid reporting “durability per kb”, which isn’t linear, and more difficult to interpret.
Honestly, the durability number is so high, that adding the bytes as a unit would just make it difficult to reason about. And S3 doesn’t store “bytes” — it stores objects (made of bytes). So, that’s the metric they chose to focus on.
* have a bad block on one drive * have a bad block in *EXACTLY SAME LOCATION* on the second drive * have a bad block in *EXACTLY SAME LOCATION* on the third drive * all of that happening within one scrub period (which is usually once a month).
And that's just simple RAID6 setup without external redundancy.
As you can see it would be far easier to lose data due to just correlated drive failures, than from random block corruption.
Backblaze takes user data, splits it into 17 pieces, add 3 pieces of redundancy, and split it across 20 racks. Backblaze has a tiered response system, my memory is something along the lines of 1 of 20 disks dying = schedule for replacement the next business day. The 2nd disk triggers a quicker/higher priority response. 3rd disk triggers pages and all hands on deck.
Given that disks have a annual failure rate on the order of 1%, and losing the first disk has a chance of 1% * 1/365, if a second disk is 4 hours that 1% * 1/(3656), and a 3rd disk is 1%/(36524) you end up with lots of zeros:
0.01(1/365) 0.01 * 1/(3656) 0.01 * 1/(365*24) .00000000000000014281
That's just to remove the redundancy, you still need to lose another disk before data is lost.
Sure there's other failure methods, including those that could take out an entire data center. I did find a statement saying that data stored in the US-West region would be stored in both Sacramento and Phoenix, but not sure if that means each B2 object uploaded would be in both.