Hacker News Reader

"The reliability under your circumstances might be wildly different."

No. I've seen this argument raised a few times whenever hard drive failure rates comparing manufacturers has occurred. (Previously on HN: https://news.ycombinator.com/item?id=7119323 ) Backblaze puts more stress on their drives than typical users, but a drive which works well in their environment is also very likely to do the same in a less stressful one. This is how accelerated life testing is done.

https://en.wikipedia.org/wiki/Accelerated_life_testing

Their pod approach even results in worse performance

Not once in the data presented in the article is performance mentioned. This is not about performance, it's about reliability. It may be anecdotal, but the numbers I see correlate well with my experience and many others' experiences I've heard of.

More amusingly, I remember coming across a forum discussing data recovery, which happened to be split by manufacturer, and noticing the Seagate forum had an overwhelmingly large number of topics containing people asking for help with their dead disks relative to the WD, Toshiba, Hitachi, etc., which was disproportionate to their marketshare.

That was my understanding too, but is it sure that there's no counter intuitive factor possible ?

well, if a vibration free environment led to failures being rare enoguh that the difference was immaterial, that would be good to know. you could save money or opt for better.perf at the same price.

At Google my colleague developed a simple test that made vibration effects very obvious (this was in 2008). In a server with a bunch of disks, do random reads or writes on all but one disk. On the final disk, do sequential reads. Back then you could get on the order of 100MB/s sequential read speed on the outer edge of a hard drive in ideal conditions. Under this test performance would dive to something like 10MB/s or worse. All of the seeking would cause vibration, which caused misreads on the disk under test. When a bad read happens a hard drive has to wait until the next rotation to try again. We eventually fixed this with better damping.

I am that colleague. While it great that Backblaze releases this information, there are many factors that you need to keep in mind when looking at this data.

With respect to vibration: we found vibration caused by adjacent drives in some of our earlier drive chassis could cause off-track writes. This will cause future reads to the data to return uncorrectable read errors. Based on Backblaze's methodology they will likely call out these drives as failed based on SMART or RAID/ReedSolomon sync errors.

HN is amazing.

Threads like this are what it's all about :)

Does Google use off the shelf drives like these guys?

I don't think I can comment on that! Sorry. Someone who is working on storage at Google today would have a better idea of what's public vs not.

I think it's been known for a while! Here's an old video of a guy inducing read latency spikes by shouting at some servers: https://www.youtube.com/watch?v=tDacjrSCeq4

"a guy" in this case being Brendan Gregg, http://www.brendangregg.com/, one of the prominent experts in server performance!

I had an argument from a user who wanted consumer drives instead of "RAID edition drivers". So I ran a test in a 1U server with 8 or so 15k rpm fans.

With consumer drive outside the case, on a bench: http://broadley.org/bill/consumer-no-vibration.png

Inside: http://broadley.org/bill/consumer.png

Server drive inside: http://broadley.org/bill/server.png

We went with the RAID edition.

>What you're seeing is not the most reliable, but the most reliable under their particular set of bad conditions. The reliability under your circumstances might be wildly different.

You do have a valid point but I also wanted to add that my harddrive failures in my gentle home environment matches BackBlaze's more hostile environment. I have 50+ harddrives in various capacities from 2TB to 10TB and my differences from BB include drives being iso-mounted on rubber grommets and they also do not run 24/7. (I only power them up when I need archived files off of them.)

Even with those stark operational differences, my harddrive failures exactly match the generalization of Backblaze's data: More failures with WD Green and Seagates especially the 3TB drives. My HGST drives purchased over the last 4 years have zero failures so far.

Curious, what are you doing with so much storage at home?

https://www.reddit.com/r/datahoarder

uhh, videos of the adult type...

> They do very little to dampen vibration amongst other things, based on their last pod design.

Yev from Backblaze -> In Storage Pod 6 we actually did a lot more to work on the dampening (and 5.0 as well, but 6.0 builds on it). The drives are all placed in to guided rows and are held down by individuals lids w/ tension on them. You can take a gander at the latest build here -> https://www.backblaze.com/blog/open-source-data-storage-serv...

Does an average server have much to dampen vibration? Something like [1]. I occasionally replace drives in similar machines, and I haven't noticed anything that looks like it helps with vibration -- but I don't look very much.

[1] http://www.dell.com/us/business/p/poweredge-r530/pd?ref=PD_O...

Average servers don't tend to have much.

Storage servers do tend to do stuff to dampen vibration, but then it's a question of storage density. When you've got something like a backblaze pod that's 60 drives in 4U of space. It'd be more unusual to have 60 drives in total in a rack full of normal servers. Now consider 60 drives / 4U, and you'll have multiple of these storage pods in a rack. It adds up.

Where can I see a public stats breakdown of similar magnitude of your preferred environment?