Hacker News Reader

Comment by beagle3 | Hacker News Reader

I still pay for snooty, and the reason for that is that when a disk goes bad (not if; when) I pop its tray out, replace the disk, pop the tray with the disk back in, click a couple of widgets, and that’s it. I know it will be rebuilt properly.

(And I know I have to do that, because when the disk fails it beeps and lights a led near the bad disk)

It’s easy to build a NAS such as the one described in this article, but in the long run, data loss is significantly more likely.

Also, any guide like this that doesn’t guide you through “disk 3 failed, this is how you safely replace it” is imho incomplete, even if it doesn’t go through telling you how you know a disk has failed.

View on HN

Is snooty a autocorrect for Synology or some other product?

Indeed. Didn’t notice and it’s too late to edit now.

That is kind of exactly how zfs works though. The guide isn't complete, sure, but "rebuilding" the array is just replace the disk and run a single zfs command.

That’s comforting to know; that wasn’t true in the past for ext4 over lvm (is it true now?).

But what is that command? And how do you know which disk has gone bad?

I am sure I can get an answer from Google / Claude / ChatGPT, but a guide is incomplete without it - and the failure report should be active like a beep or flashing hardware light - I typically log into my NAS only a few times a year. A motd or other banner isn’t sufficient.

zpool-replace(8)

https://openzfs.github.io/openzfs-docs/man/master/8/zpool-re...

ZFS got this right. MDADM on ext4 somehow did not.

EDIT: ext4 on MDADM.

How ZFS gets this right: zfs(8) on zpool(8). So, basically, ext4 and mdadm can talk to each other.

> in the past for ext4 over lvm

I used to look after storage arrays for VFX places.

everytime I saw lvm I inwardly sighed. The docs were terrible, almost as bad as MDADM. snapshots were for a long time unrecoverable. You'd then have to work out what pattern of LV you had, was it a suprise raid0? or a misaligned raid1?

zfs is a night and day compared to LVM/mdadm, two tools, rich help, the man pages are reasonably good, and once you understand zfs vs zpool, you're usually good to go.

man zfs, man zpool, cron, your mta of choice (I like dma) :)

schedule a zpool scrub every month, send an email if it finds errors. zpool status will show the errored drive. zpool replace will initiate the drive replacement.

ZFS handles drive failures more robustly than anything else. There's a reason synology uses mdadm under BTRFS instead of the built-in BTRFS RAID features, and mdadm operates at the device level. That means that to replace a drive, mdadm has to rebuild the entire drive while zfs will only rebuild what's actually in use.

> Also, any guide like this that doesn’t guide you through “disk 3 failed, this is how you safely replace it” is imho incomplete, even if it doesn’t go through telling you how you know a disk has failed.

`zpool replace my_pool disk3 newdisk`

The worst part of either approach is: how do I find disk number 39 out of 48.

By using /dev/disk/by-id when adding them to the pool - and looking at the physical label on the disk itself. All disks I ever worked with had that ID printed somewhere on there.

ZFS makes this completely trivial except for the "beeps and lights a led" part

To be fair, the "beeps and lights a led" part is very important in case of a dead disk in an array. The only time, I disrupted a service in production for longer than a few hours was when I didn't get the email of a degraded raid 1 array and the second disk died a few weeks later.

smartmontools makes this trivial and can email you which might be better than beeping and lights if you’re not always near your server.

I use that plus smartctl_exporter so I can do metrics and alerting off that.

That’s awesome. But a guide that does not include a howto for alerting (such as the one linked) is incomplete.

smartmontools misses data corruption at the drive layer, as it trusts the drives to report faithfully.

"In hashes I trust"

Does Synology with its beeps and bops catch that?

Yeah, for that I use periodic scrubs and zfs event daemon

Reply on news.ycombinator.com