Hacker News Reader

Comment by tomas789 | Hacker News Reader

For years I’m running home server. Mostly for storage plus some side projects. I only had a little time for server maintenance and I ended up redoing the whole server every time a disk failed. I was using just regular desktop grade disks so that happened every 18 months or so.

This all changed when I started using ZFS. Not only it has support for raidz and mirroring (which I could get with LVM too) but it is tweekable and tunable easily. Plus commands like _zpool status_ will give you a great overview of the health of the array in no time. It might seem like nothing but it makes all the difference for me.

I can recommand for everybody running a (home) server. It will save you lots of time.

View on HN

What kind of disks and brand do you use? Are you doing some kind of disk-intensive work?

I am asking because this sounds really short to me and I am wondering if I am extremely lucky never to have a failed disk in more than 25 years, mostly with more than 3 computers at a time with only consumer grade disk of random brands. I only swich them because for better performance/capacity, but some of them are almost 10 years old.

I still have a 1.6gb IDE drive that is functional. I have drives with 70k hours that still run. I've had 2 drives fail in the last 30 years. It may be rare but when it happens you'll really wish you had a backup.

Like my fellow commenter, I also do have a few drives with over 50k head hours on them that still work fine, but I also had probably more than half a dozen drives fail on me or break (not including Sandforce SSDs, that'd be unfair), including one that had all of my early projects on it. Really wish I had a backup of that.

Now I have a backup of everything and whenever I have to think about how much storage I need for something I always take that number and multiply it by three -- if it's only stored once or twice, it really can't be more important than /tmp.

Interesting for me, I’ve had two drive failures, both in the past year, but otherwise nothing for the past five+ years. Both were data center class drives (Seagate Exios) and died within the first 2 months of purchasing them. I have over 20 consumer grade drives with >50,000hrs of reported power on time, of which 3-4 are >70,000hrs. They constantly see writes, but I’m definitely not pushing max IOPs by no means on a regular basis. I’d say around 1-2% of the time they are under very heavy read/write activity, otherwise just constant appending of data at around 50Mb/sec across the entire RAID pool.

This is a pretty common pattern with disk failures - in my experience they either tend to fail in the first few months, or they make it for the duration.

It's even got a name: the bathtub curve[1].

[1]: http://www.applied-statistics.org/Glossary/BathTubCurve.html

Use Backblaze's hard drive stats as a guide: https://www.backblaze.com/b2/hard-drive-test-data.html

I've yet to experience a failure of HGST drives, even after 10 years of operation in an LVM or ZFS array.

HGST remains my preference for "enterprise-y" stuff on home and small business setups. Slightly pricier but if you value your time it pays for itself easily.

I buy used 4TB HGST drives on eBay. They’ve all been solid and inexpensive, some with more wear and tear than the others.

I have a couple I'm looking to sell. Send me an email for the details.

Are you sure your data as stayed uncorrupted all this time? Another thing ZFS enables is detecting bitrot and corruptions (happening due to unexpected power loss or otherwise) that you otherwise may have been undetected but can still cause issues over time.

I was thinking the same thing. I still have two 2TB Ultrastar's that I bought re-furbished 5 years ago, and which have been running in various arrays for just about the entire time since then. knock on wood

https://www.backblaze.com/blog/backblaze-hard-drive-stats-q2... has the best statistics that I know of. Basically a 1% failure rate each year (worse for a few bad batches of disks), so yes I think you have gotten lucky so far. (1 - .99^(25*3)) = 52% chance of at least one failure for three drives over 25 years.

Out of ~30 disks over my ~30 years of hard disks I can count at least 3 5.5" HDD failures and 5 or 6 2.5" HDD failures. Laptop drives seem to fail at a much higher rate especially for kids.

You are lucky, or bought the right brand of drives at the right time.

I had five new Seagate 7200.11s which had an easy life but not make it past 22 to 25k hours before they started failing en masse (and thats not even counting their firmware bug that got me). ZFS RAIDZ1 (RAID5) pool, which survived a second drive starting to fail while rebuilding from the first failure. It was that event that made me love ZFS forever.

Contrast to my WD Reds which are 52k+ hours without errors or issues (no failures in a set of 8). And some HGST refurb drives that are at 70k+ hours (some failures in a batch of 14, but they were refurb with wiped SMART data so the failures werent unexpected).

Agreed. I've been aware of and thinking to check out ZFS for over a decade because all these weird nerds keep talking about it but it seemed so involved and complicated.

One simple command each to:

* create a zpool

* create a dataset (persistently mounted across reboots, with optional encryption and compression etc)

While, yes, properly tweaking ZFS for performance and making proper decisions on things like recordsizes, L2ARC and SLOG requires a bit of deeper understanding of how it works, the CLI is very approachable and the man pages are straight-forward and easy to understand for a beginner.

Compared the alternatives of cryptsetup/mdadm/lvm/lvcache, it's such a breath of fresh air and a lot easier to work with. It's unified and intuitive and things just make sense.

The big game-changer for me is its caching mechanisms. For workloads that already have efficient caching (mature databases, for example), it might not make a big difference, but for things doing redundant IO it can improve the performance by an order of magnitude in a way the Linux page cache just can't.

Put a fast NVMe as L2ARC in front of a mirror of slow but inexpensive disks and you can eat the cake and have it.

Oh, and native compression (currently lz4 but zstd is coming) that barely impacts performance (and sometimes improves it).

I'm one of those weird guys now and even my nerd friends don't get me anymore :')

Not to take away from the merits of ZFS, but this sounds like something that 2 disks with software RAID1 would prevent from happening.

Also, 18 months sounds kind of short, I usually get around 5 years of use from hard drives. What brand are you using? I highly recommend checking out the quarterly Backblaze HDD failure reports.

Source: been running a server (read: consumer headless desktop) for years without issues.

More than half the time in my experience it’s usually a failing sata cable or power-supply if the hard drive continues to functions on boot but eventually goes to degraded due to checksum/read errors. Sometimes the data is just fine but a read error occurs between the drive and controller which ZFS interprets as a failng disk

modern drives have onboard diagnostics which can report errors directly to the OS. I'd be very surprised if a modern file system was unaware of this. I'd also be surprised if users of alternative file systems didn't test a suspected failing drive before concluding it was faulty

The error is occurring between the drive and the sata controller on the mobo. There’s also the whole issue of whether the OS or the drive is responsible for error handling. If the the drives are SAS drives, typically the OS uses fire and forget methodologies for write commands. Other drives might not support that and require zfs to handle errors at the software/kernel level

replacing the cable / testing the disk on an isolated system would confirm this

It would but I encountered a situation once, where when I would remove the drives of the array and test the disk 1 by 1 they would all work. It wasn't until all the drives were powered on and under max write stress would the power-supply under-volt to drives randomly making it appear as multiple drives were all failing.

the great thing about zfs is you don't have to care, at all

as long as it writes and reads back (most of the time): ZFS will deal with it

Hard drives lie. SMART data is not thorough and it is not trustworthy. Across dozens of drive failures over the years, I've never seen SMART predict a failure before ZFS detected data loss or the OS reported 'drive is slow'. SMART is not worth the effort to monitor.

Filesystems don't read SMART data. You might have a separate daemon which monitors SMART.

ZFS checksumming is amazing for this. You know, without doubt, which file(s) are bad. You can still use failing or unknown quality drives because the checksumming will protect you from silent data corruption.

SMART marking drives as failed is usually super late. But if you monitor the raw values for the important parameters (unreadable, reallocated, etc sectors), I've had good luck with replacing drives before software notices. At least for spinning drives; SSD failues were way more rare, but resulted in the drives completely disappearing from the bus, and no reliable prefail indicators. I did have one SSD go through a big reallocation that tanked throughput and the alerts from throughput and the alerts from SMART thresholds fired simultaneously.

Of course, that's great for a server farm; in home use monitoring is a lot less structured.

Expensive enterprise drives are generally better at smart stats. Of course your mileage varies by vendor. Desktop drives on the other hand are completely untrustworthy.

One pro of ZFS over RAID1 is that ZFS is file based and not volume based, so in case of recreating mirror disks (or general resilvering) you only copy data, not the whole volume.

> Not to take away from the merits of ZFS, but this sounds like something that 2 disks with software RAID1 would prevent from happening.

You're not wrong. And if there's a mechanical issue with a drive and it dies / stops spinning, then you'll probably get an alert and can swap it.

But if there's any kind of bit rot or data corruption, and it only happens on one side of the (traditional) RAID1 mirror, how will you (a) know that it actually occurred, or (b) know which side is good bits and which side has bad bits?

With ZFS and checksums, you can be confident that your data is still healthy. Depending on the data, this may be an important consideration.

zfs send-recv is also very handy for doing incremental backups.

What ZFS does here is not enough for me. I need to be alerted to changes done also above the file system layer, like by malware or accidental deletion etc. Only way I have found to solve that is to have checksum tool above the file system. If something is wrong, I restore from backup, so zfs does not give me anything...

I don't personally do this, but if you're taking scheduled snapshots of your ZFS filesystems, you could also have a scheduled job (say, nightly) that emails you a "zfs diff" [0] between the current snapshot and the one from 24 hours ago. It won't tell you that you've been hit by malware, but an unexpected spike in changes could be something worth investigating.

[0] - https://docs.oracle.com/cd/E36784_01/html/E36835/gkkqz.html

You can do a zfs diff command on two snapshots or a snapshot and current dataset as zfs takes a just-in-time snapshot of the live dataset.

> I need to be alerted to changes done also above the file system layer [...]

As others have mentioned, would "zfs diff ..." be useful?

* https://www.thegeekdiary.com/how-to-identify-zfs-snapshot-di...

As the name suggests, "snapshots" are read-only and so cannot be altered. You could either copy/rsync the modified file/s to the live location, or do a rollback to a particular snapshot:

* https://www.thegeekdiary.com/solaris-zfs-how-to-create-renam...

If the machine is compromised in some way, you could reinstall and do a "zfs send-recv" of a pool from a remote system.

You can clone the snapshot (so its clone become writable, not the snapshot itself!) and you even can promote a snapshot to a "parent" filesystem (reverse parent-child).

Strange. I have some WD disks in my qnap running 24/7 for 8 years now, without a single problem. I'm planning on reusing them in my new NAS..

I have a WD Purple I bought in 2014 and it's been running continuously since. 18 months seems extremely short.

Isn't Purple for things like camera feeds? Good on writes, bad on reads.

This seems to be the conventional wisdom but I can't say I've experienced it in practice. They have less cache than Reds and the firmware may be tuned for writes but I think it's mostly down to different market positioning/branding & price. You can compare the two: https://documents.westerndigital.com/content/dam/doc-library... and https://documents.westerndigital.com/content/dam/doc-library...

Same here, I have over a dozen WD Red drives with over 70,000 power on hours and all pass their SMART tests.

> I was using just regular desktop grade disks so that happened every 18 months or so.

That seems unusually bad. Up until earlier this month my old raid array has been mostly desktop drives(just upgraded it to IronWolf NAS drives though). I did finally do the upgrade to new drives because one died, but that drive was manufactured in 2011, and the rest of the drives are from 2010, 2013, and 2015. To be fair I've not gone five years with out a failure, I think the 2015 drive was added about 3 years ago(being my actual desktop drive before that), but otherwise have had good luck. In fact, I think my most recent failure may have been accelerated by my server being put into a temporary case with poor air flow.

> so that happened every 18 months or so

Might be power supply issue.

I like to separate storage from the rest. Had various models of synology for nearly 10 years now and it’s the ideal solution for stopping to think and worry about it (combined with UPS). Running VMs, etc, I’d rather do that on a different box. You need to reboot frequently, make some experimental changes, modify the hardware, etc. Better to do that away from your data.

Reply on news.ycombinator.com