Hacker News Reader

Comment by raverbashing | Hacker News Reader

I usually find people running ZFS to run into way too many more problems than what other people run using Ext4. (Edit: I'm surprised to see how people run into issues with Btrfs)

It might be good for some specific purposes but I'd reckon the majority of people using it do not fall into those. Using it "just because" is probably more trouble than worth.

> ZFS really wasn't in use, ever! But at the same time, it was eating over 30% of CPU capacity! Whaaat??

Interesting bug, sounds like the best option is to not even have the module loaded then. (Not blaming the developers here, necessarily, but yeah, it's a bug)

View on HN

Usually in my experience whenever someone has a problem with zfs it’s solvable in one way or another and the devs are very responsive on irc and GitHub.

On the other hand a lot of problems caused by btrfs is unrecoverable and a mess. Ext4 is stable but lacks the feature set that makes customers go to zfs instead (and that is fine, software should do one thing well and right).

Btrfs gave me the most trouble by far. And most people say the same in my experience. Replicating half of ZFS's features with ext4 is a whole lot of trouble too.

We have a machine that has a large ZFS storage pool, and one smaller btrfs volume on a pair of SSDs. If I delete a large directory hierarchy on the btrfs volume and then run sync, it locks up the entire machine for 5-10 minutes. WTF. Nothing like that ever happens with ZFS or ext4.

I've had a lot to learn with zfs, but it's never just flat out lost my data, unlike btrfs (in about 2017)

Raid5?

Not sure if you are suggesting RAID5 as a relevant problem or solution, but it's arguably the former! :)

Regardless, ZFS has configurations equivalent to most "traditional" RAID options anyway (raidz1 is their RAID5) but with the added benefits of all the other great stuff ZFS gives you, like checksums, compression, deduplication, snapshots, etc.

Indeed the read5 implementation was (is?) notorious for some time (or still), well if you Google you'll soon find the stories.

Probably it was a guess that you were using a RAID5 mode of Btrfs, and that it was the reason for the failure.

Not using RAID5 or RAID6 modes on Btrfs is the first piece of advice people considering Btrfs get.

You are now even warned by mkfs.btrfs itself when you attempt to do it. [0]

[0]: https://www.phoronix.com/scan.php?page=news_item&px=Btrfs-Wa...

Right, I failed to parse the comment as inquiring as to if that's the Btrfs issue referred to. Yes, I am familiar with Btrfs' RAID blunder. Somewhat ironic though since RAID5 is rarely the best idea regardless of the system implementing it

Consider yourself lucky. There are data-corruption bugs all over the now-diverged ZFS codebase(s). Check the Solaris 11 SRU notes from the last 24 months for many examples.

For a homelab use-case, I backed out of creating a ZFS mirrored storage pool for network storage. It wasn't that the steps were too complicated or that ZFS let me down as such.

Just that RAID in general and ZFS put my files in a blackbox, and that I didn't really need a significant boost in network reads. I settled for plain old ext4 and periodic rsync mirror files to another disk (I don't mind some interim data loss + writes are fast). Use SFTP or sshfs for accessing the drive.

Any RAID solution that does block/bit level striping will be a “blackbox” as you describe it. That’s not a bad thing (nor is your approach) but pros and cons.

Yup, and as far as "blackbox" solutions go, if something horrible should happen to your system, you want something that works like ZFS as opposed to, say, some high-performance but proprietary hardware RAID.

As someone who has a couple times needed to rip 100tb of ZFS disks out of something, put them into a different machine with a different architecture or OS in some random order, and access everything without having lost anything, it's hard for me to overstate how great it is that such a thing is even possible.

How exactly did you do that? I'm learning ZFS still and I'm planning to move some of the disks in my first NAS to a better machine.

Which ZFS commands?

Nothing special is necessary. All you need to do is import the pool as normal.

There’s a potential problem, in that you can’t import a poll on an OS version that lacks the feature flags that are enabled on the pool. The way to solve that is to choose a common subset when creating the pool.

Unless they fixed it, another edge case is a change in processor endianness. I started working on a patch a long time ago, but it necessitated changes to code that had remained unchanged since the original release... not a blamelog I'm totally comfortable with.

I hadn’t heard of that one. Does send |receive still work?

Nope, the system would instantly panic - presumably in order to protect data integrity... imagine the damage possible otherwise. It was actually a fairly ponderous debugging effort, as I remember it, because it was occurring on an already shaky POWER9 setup I was using to port other stuff. ZFS worked great for any vdev setup on any OS compiled with the same endianness, but plugging in so much as a thumb drive previously provisioned from a different endian host... insta-console-vomit. Definitely an uncommon problem though :)

What?

CPU endianness switches work fine, IME - I have pools I originated on SPARC Solaris that work fine on x86, I just got a patch landed for an edge case in one recent feature not interoperating properly between endiannesses but other than that it's worked fine for me.

What kind of issue did you see?

> CPU endianness switches work fine

Thats great news, hopefully that rare potential problem got addressed in the last year or so.

Steps to recreate: compile linux (maybe FreeBSD as well, that is a little more fuzzy) for LE. Boot new kernel and setup a new pool. Recompile for BE. Boot new kernel and enjoy bootloop...

lol, actually things are starting to come more into focus... page size played a role in the whole thing. As I said, edge case :) No potential for data loss though.

I'm guessing this was probably PPC, given the rarity of the other options?

Sounds like a fun experiment to run down. Maybe I'll go look when I'm done the current nightmare I'm tinkering with.

Yup, Raptor Blackbird. I'd send along my notes if I didn't think they'd be more misleading than helpful. Left that one in a mess that'd be difficult to untangle, but I remember the trail basically ending in a file that was practically unchanged since the original source got released. Because of the code vintage I really wanted to be sure that I didn't introduce performance regressions, so that led to me turning my attention to live kernel debugging over serial, which led to uncovering an even more ancient tty bug (seriously, we're talking unix fountainhead old).

She swallowed the spider to catch the fly... I'm presently writing something to crack 8051 firmware xnor'd by a 64 byte key, I don't remember how - but the previously described ZFS edge case somehow got me to this point.

I am...familiar with falling down rabbit holes.

I cut a patch to fix sparc64 building with the new zstd feature. I then discovered the new zstd feature was broken for endian portability because lol bitfields.

`zpool export` on the machine your taking the array out of and `zpool import <pool name>` on the receiving end will do the job in most cases.

Hopefully, the pool was exported before you moved the disks. I came very close to losing data (but ultimately didn't) when I yanked a pool out of an OpenIndiana system and tried to import it on Ubuntu without exporting it first. ZFS really didn't like finding the disks on devices with radically different names than where it had left off.

It ended up re-importing OK when I brought it back to the OpenIndiana box. Whew! I then did an export, and Ubuntu was then able to import the pool.

When you import without exporting, all you risk losing is the last few seconds of data transactions. If a motherboard dying was enough to lose all your data people wouldn't really recommend ZFS.

So hopefully the pool was exported, but the worst case scenario isn't very bad.

I make my pools with the serial number /dev/disk/by-id/ names, like ata-WDC_WD30EFRX-68EUZN0_WD-XXXXXXXXXXX.

That is not going to change between different machines as far as I know, I might be wrong about this. I made some bad choices (in hindsight) when making this pool for my uses, but it's still going strong many years later. [1]

This current pool (which is now old, 50k+ power on hours on these disk) has survived a motherboard dying randomly and one or two SSD failures with the OS on.

Always just installed a OS on a new SSD and it's been picked up just fine.

[1] I went with RAID-Z2 for 6x3TB WD Reds where I probably should have made mirrored pools or something like it to gain more space? It's been a while since I looked at it. Can't really expand this pool or add more storage without replacing each disk one by one with something bigger.

I could make another pool with new disks but I'd lose another 20% to parity.

I use those names as well, at least now that I entirely use ZoL. That wasn't an option back in the day with OpenSolaris and Illumos. You'd be using names like c0t0d0 instead, and woe to you if it found the wrong disks in the wrong places (it would just refuse to import claiming the pool was faulted).

Caveat: things may have changed since then - it was at least six years ago.

I have only used ZoL myself, on Debian and Ubuntu. I really sorta need to buy new disks for this, but they are so expensive right now. I should probably just backup all the stuff I really care about and offload most of this to some sort of cloud storage/rented space somewhere. I don't know that these drives will fail (WD Red 3TB), but the 50k+ power on hours are starting to worry me a bit.

They have not seen a huge amount of reads/writes though, if we don't count the weekly scrub and weekly usage by just me.

> I settled for plain old ext4 and periodic rsync mirror files to another disk

This isn't a mirror, not in the sense that most would understand. No performance boost, no protection from bit rot, unnecessarily wasteful in every potential metric. If you wanted to intentionally design a system to propagate errors and render backups useless - you would start with this kind of setup. It makes sense to avoid all the RAID related problems associated with hardware solutions... but I'm drawing a blank on rational reasons to do what you've described. Super weird boot manager + physical space constraints?

Two standard external USB disks of different ages, connected to a powered USB hub attached to a ROC-Raspberry Pi. The older one is the "live" one that serves all data requests. The newer disk is usually attached to the hub, but it's also disconnected during summer.

If the "live" disk fails, the other disk will eventually replace it.

> render backups useless

Like I mentioned, data loss isn't a concern. This is mainly hosting code repositories and media files.

What's my use-case? My laptop, that has a paltry 256GB SSD, keeps running out of disk space, and I often find myself plonking files that aren't super critical on the network drive.

> but it's also disconnected during summer.

I knew there was a reason I was getting a "can't send mail more than 500 miles" vibe... Are we talking small form factor spinning rust, or thumb drives? Because if you thought that ZFS was a black box - look into flash wear leveling algorithms, I'm hard pressed to think of any storage media more prone to spontaneous bitflips. Unless you are checksuming before and after every movement of data - your files are going to silently get corrupted in ways you won't notice, until something breaks.

Anyway, I've had way more instances of bitflips than drive failures - and I've had to deal with that kind of data corruption escaping detection and making it into backup archives. That is why I obnoxiously promote ZFS to the degree I do... it totally eliminates that risk automatically. No joke, this thread prompted me to run a manual pool scrub a month in advance for an 8TB archive pool. Since it is archival, it gets few reads - which means it gets few automatic checksum verifications outside of the scheduled bi-monthly scrub. Well the scrub is only half way through, but it already shows that at some point in the last month a 128KB block on a single 3.5" disk got silently corrupted. Data corruption nipped in the bud, thanks to ZFS <insert whatever Sun's jingle was>.

Again, you're focusing on data consistency, whereas I'm not really concerned about a bunch of cloned throwaway OSS git repositories or npm modules or MP3 files being corrupt over time.

These are old WD Passport spinning hard-disks that were never designed or optimized to run as NAS drives. I had them lying around doing nothing. This is a salvage operation for the old drive.

> Again, you're focusing on data consistency...

All that talk of "homelab use-case", "mirrored storage", "RAID", "rsync"... obviously what is under discussion is how ZFS is a poor fit for the tmpfs tier garbage data use-case, dunno how I missed it.

Reply on news.ycombinator.com