I failed statistics. If out of a million hard drives 5000 die in a year and take 15 minutes to swap, what are the odds of 2 failing on the same machine?
If in a year out of a million hard drives only 5000 die, then you're projecting a 200 year average lifetime per disk drive. No real disk has that. A more reasonable 5 year average lifespan gives you 200,000 failures per year. Which is much worse.
Next, you're asking about the odds of 2 failing on the same machine. How many disks are on a machine? 1? 10? 100? Are failures independent events? It makes a huge difference. In fact they are not independent because when the motherboard craps out you lose access to all disks on that machine at once. At their scale it is too much work to figure out whether some of that data is recoverable - you just assume there is another copy somewhere and throw away the stale data. If you're wrong, then oops.
You are also throwing out the 15 minute disk replacement time. It may take 15 minutes to replace a disk, but that figure is irrelevant. To replace a disk you have to locate the machine, and it has to matter enough to you to send a person out. I guarantee you that the time before a person gets involved is going to average more than 15 minutes. Generally a lot more than 15 minutes. (Google famously takes the attitude that it is generally more work than it is worth to find the broken machine, and lets most dead machines sit there indefinitely. I wouldn't be surprised if other cloud providers imitate this.)
Next you have to consider that the end user shouldn't care about machines. For the purpose of redundancy Amazon is not going to keep multiple copies of the same data on the same machine. They are going to put them in different machines, and hopefully in different places. That will reduce the odds of a single failure losing your data.
All of that said I am somewhat shocked that Amazon would advertise a 0.5-0.1% rate of data loss as acceptable. I don't know Google's actual failure rate, but I'd be willing to bet large amounts of money that it is much lower than that.
For instance search for "gmail lost data". The only significant gmail data loss that turns up was in 2006. (See http://techcrunch.com/2006/12/28/gmail-disaster-reports-of-m... for more.) A grand total of 60 accounts got wiped out. Subsequently most of the lost data was restored from backup. (I doubt that the error was at the data storage layer.)
That's not just better than what Amazon delivers. That is ridiculously better.
Amazon has said they've never heard of anyone experiencing data loss on S3. So if you use EC2+EBS+S3 properly, you should not ever experience data loss except data captured since your last snapshot to S3 (which should be extremely frequently).
Every single drive in the datacenter typically has the same likelihood of failure (since they're usually the same make and model, and similar production runs), so the odds that a drive somewhere in a large data center is failing RIGHT NOW are in reality rather high.
The Big Mac ended up being a good example of this. The likelihood of having a memory error on a normal PC is low, because the likelihood of a single DIMM having a memory error is low.
Punch that up to the 22,000 or thereabouts DIMMs that populated the Big Mac cluster, and now you're looking at a very HIGH likelihood that a large-scale computation using the cluster will experience memory errors, and therefore produce invalid results -- which is why UVa ended up replacing the entire cluster with ECC-equipped machines in short order. Until they did that, researchers had to run simulations multiple times and compare results to make sure that their simulations weren't contaminated by memory errors.
No thats the point, hardware failures on the same machine/rack/psu/site are not independant. A power supply spike that kills drive 0 in a RAID will probably kill the mirror drive 1 as well - that's why RAID isn't a bckup stratergy
I was explaining a common misconception about statistics, and using hard drives and DIMMs as examples.