Hacker News Reader

Comment by gruez | Hacker News Reader

>In other news, all of the WDRed CMR drives in my ZFS array started throwing read errors simultaneously so I've ripped them all out.

I don't get it, is the implication that them showing errors at time should be a freak occurrence, and such an occurrence means that WD drives are low quality? Drive failures are more correlated than you think. If you bought and installed the drives at the same time, such a failure case is totally expected. The drives probably came from the same batch, and experienced similar amount of wear, so it's only logical that they experienced the same failure case.

View on HN

Ah, the RAID array love story: one of many similar disks fails, you replace it, array starts rebuilding which causes an increase of reads from other drives and before rebuild is complete, more disks from the same batch start failing.

I don't deal with storage hardware much for some 6+ years, since I've switched to cloudy solutions. But at the beginning of my shared storage building journey, I considered arrays built from different disks somewhat "lame" or "unprofessional". I changed this viewpoint 180° after seeing multiple similar drives fail in a very short (hours, days) timespan for the second time.

I now have more trust for home-built ZFS setup using different disks than a dedicated storage appliance using enterprise-grade disks with almost following serial numbers.

Raid array should always be scrubbed every few months to prevent this exact situation

Scrubbed as in erased? Where do you temporarily place 1PB+ of data while that’s happening?

`zpool scrub $Name_Of_zfs_Storage_Pool`

https://docs.oracle.com/cd/E18752_01/html/819-5461/gbbwa.htm...

Scrubbed, as in performing a periodic read of the entire array to 1) check the data for consistency and 2) exercise the drives.

Scrubbing helps fix random errors in the drive data, but it won't prevent correlated disk failure. If a drive fails during the scrub, then you still have to replace the disk, and it's still very likely that another disk can fail in that time. If you just want to eliminate lemons, then you can do a burn-in test before adding it to your array, but IMO even bad disks usually don't fail that quickly.

I guess so! I was speaking with someone on Mastodon today with some WD Reds and they failed at almost exactly the same number of power on hours as mine (~43k hours).

Given mine were in a ZFS mirror, they would have had almost identical wear so I suppose we could say they that they're predictable (though mine are the CMR drives which you don't get in the 4T these days and SMR drives are bad for ZFS).

If I was to replace them with the same drives I'd have a rough idea when to replace them before they start failing but I'll be buying larger capacities these days so those numbers are probably out the window.

These are spinning disks, not SSDs. Spinning disks are not known to fail like this, unless the manufacturer made a horrible mistake.

On the contrary. The same principle applies to them. Spinning disks might not have NAND cells that wear out, but they have other parts that have manufacturing defects and can experience wear: bearings, actuators, heads, etc. If you have 100 identical cars that rolled off the assembly line in sequence, you would their failure date to be be clustered, rather than being evenly distributed.

On a tangent, wouldn't it make it safer if you didn't start a NAS with a bunch of identical drives, but you got them from different manufacturers and models?

They'll still fail, but hopefully they'd be less likely to do it at the same time.

Yes, that’s generally good advice.

this is news to me and taken away my excitement of having my first nas. it's very good to know though. i wonder about good-guy computer stores, mixing batches for naive customers like me, ordering 2 identical drives thinking it was best. it's crazy, it really defeats the purpose of raid 1 if both drives are likely to fail at the same time. i guess i should buy another different brand drive, and keep one of the current pair in a drawer as the future replacement.

To take some further wind out, with current disk sizes there’s a decent chance of a second disk failing while you’re still replacing the first. Reading all the data off a disk is a stressful operation, you see.

But that means you need RAIDZ2 or erasure coding to be reasonably safe, which takes you well outside what most of these turnkey systems can handle.

maybe i'll start a data-recovery savings account instead.. for whenever it happens

Backblaze works well enough.

SSDs in raids famously all failed at once. Spinning disks basically don't, outside of (typically) firmware faults. Out of 100000 spinning disks I've managed, I've never had them fail at the same time.

Mechanical things don't fail as evenly as digital things.

As someone that has set up a lot of NAS devices, we're talking thousands of disks over the years, I would not put my money on your statement.

There are a lot of simultaneous failure modes. HP had one in SMART firmware that killed disks at some fixed number of hours.

There are also temperature excursion events that can lead to arrays failing within 10s of hours of each other, not enough time to do a full rebuild of a single disk.

Seagate had really crappy 3/4 tb disks that loved to fail rapidly and nearly all at once.

Intel also had a famous bug / 'feature' where the SSD refuses to respond after a fixed number of writes.

If all your drives were Intel and part of the same array, they'd all fail at exactly the same time.

Reliable storage is hard.

So your examples of failures include a firmware example, which I said "outside of", and an example where the drives were abused?

And another poster os talking about SSDs?

Sigh

Eh? I've been admining raid arrays since before SSDs were a thing. Making sure you build your array from disks from different batches to reduce the likelihood of simultaneously failing has always been the advice.

I make no assertion as to WD quality, but failures like you describe have been the bane of sysadmins quite some time.

Spinning disks absolutely do fail this way. Drives from the same batch can easily work 5+ years fine and then fail within a few hours of each other. Even worse, in a RAID the additional load from starting a new replication after a hot swap can push a borderline drive over the edge, and this is precisely the time when your data is at its most vulnerable.

I've worked at places where they always maintained a sizable inventory of new disks so that every time a new machine was provisioned, it would receive a RAID comprising of at least 3 disks bought from different batches at least a couple of months apart from each other, and as much as possible from different suppliers.

Lol.

It happens. On my first job a batch of drives started failing daily, in sequential order by serial number. There was a bearing defect - the vendor ended up replacing about 400 drives, which meant many weekends of migration and restores.

Sounds like the exception that makes the rule.

Just because you were in a plane crash, doesn't mean it's common or frequent.

(With an apology to that guy who was hit with a nuclear weapon twice)

It happens. That scale is rare, but i can think of 5-6 times that I'm aware of where vendors proactively replace many disks.

ZFS is very sensitive and takes a long time to rebuild. I only choose RAID-10 for this reason, because I'd rather have most of my data than none of it.

Reply on news.ycombinator.com