I look forward to these reports with the same zeal and enthusiasm as I used to reserve for comic strips in Sunday newspapers.
This data is significant and appreciated. Thank-you so much, and please keep up the good work.
This data is significant and appreciated. Thank-you so much, and please keep up the good work.
It certainly is useful and interesting.
> I'd guess this is an effective piece of marketing.
Yes, it really has been good to us. :-)
It is data we would collect internally for our own decision making and tracking even if we didn't release it, and it is only a small amount of work for us (mainly for Andy the author of that blog post) to format it up every 3 months and write some observations down. Since it isn't what we "sell", it would have just gone to waste being hidden. And it results in our name getting "out there" and inevitably every quarter somebody asks "hey, what do these people do that requires 130,000 hard drives anyway?" and then we get a little bump in customers. Cost vs. benefit has been WELL worth it to us.
Along the way (unrelated to making money) it makes us happy that people get some use out of it. Like we find out somebody does their PhD thesis on the raw SMART numbers of the drives or something, we enjoy hearing about it.
Until my dying day I will never understand why Google Storage and Amazon S3 and Microsoft Azure don't release their drive failure statistics. I just don't get it. Those companies have SO MANY DRIVES and they employ people with PhDs in statistics, they MUST have the same info internally but at 100x the scale. I don't get why they don't release the numbers?! But hey, if they want to continue to give us this marketing gift of exclusivity, I'll take it. Maybe they like us and are just doing us a solid.
We had a lower failure rate across the board, for all components. I think the only protection that was in place was some blue tarps and a chain link fence.
I can't find a public reference, sorry.
Could be that on that scale they're dealing more directly with manufacturers, and they fear sharing failure data would affect it negatively?
I mean "doing it well for an influencer and shafting everyone else" is far cheaper and easier than just making a solid and reliable product consistently. That's such old-fashioned thinking these days. :)
I don't know if they're as budget-conscious as they used to be, but Backblaze used to "shuck" drives from externals because they're cheaper [1].
If this is still the case, then their stats should be similar to hobbyists who are doing this too.
> While the shucked drives might not have had the warranties
We haven't shucked drives in a while. The main advantage was when this artificial price difference that occurred in the Thailand drive crisis where a "raw drive" cost $400, but if it was put inside a $7 USB enclosure the same drive including the USB enclosure suddenly only cost $200. At the time, our business model literally wouldn't survive the cost increase, it was shuck drives or go out of business. :-) We assume the issue was that mostly "rich companies" bought "raw drives" and could afford the increase in price, but "poor consumers" bought drives in USB enclosures and couldn't afford the increase.
We still watch the prices though. You can think of it this way: if 3% of the drives fail but don't have a warranty because they were shucked, then we have to save more than 3% (plus the cost of the "shucking" process) to make it worth it. That rarely happens anymore. I think partly because as the volume of drives that we purchase has gone up we have the ability to get better "bulk discounts" than we had in the earliest days. Also, the process of "drive farming" where the retail stores would limit it to "2 per customer" or whatever would get much harder at our current scale.
Exactly, as we just saw with the Western Digital debacle.
The unfortunate reality is, only a fortuneteller can say what the best hard drive to buy right now is.
> Our failure rates were consistently much higher compared to Backblaze
That's interesting.
> The overall workload on the clusters was extremely heavy
That is the most likely explanation. We see higher failure rates when we are writing to the drives, for example as a new vault fills with customer data. Backup (our oldest business) has a fairly easy work load, it is not as punishing as a database that is pummeling the drives.
In certain circumstances when we are down more than 1 drive in a 20 drive Reed Solomon group we stop putting new data on that drive group until the parity is restored explicitly because this lowers the chances of an additional drive failing in the group. That gives us more time to rebuild the parity with less stress in our lives. When that last parity drive fails and one more drive failure means customers lose data the fun drains right out of this job. Red alerts are thrown, pagers go off in the middle of the night, datacenter techs start driving towards the datacenter at 3am to replace drives. We prefer the world nice and calm and relaxed with a good night's sleep.
/me goes and checks BackBlaze's careers page...
(I suspect driving time to any of their datacenters from Sydney puts me out of the running... At least their 3am emergencies would be a much more reasonable 8pm emergency from here.)
A particular vertical-axis wind turbine project was destroyed by buildup of magnetic dust on the generator magnets.
I'm not saying that's what was killing your drives directly. Normally HDDs are fully sealed. But the amount of dust you mention is awfully suspicious. It might change your investigative perspective a bit when you consider that some small percentage of all that dust is ferromagnetic.