It's fairly crash resistant, has well worn and battle-tested codepaths and generally seems to not loose data outside bitrot. So why pick btrfs with it's dataloss stories or ZFS with it's complex management?
It's an effort to switch to a new FS as default, especially if either itself or the integration is not as well tested or completed yet.
So everyone just sticks to the thing they know how it sucks.
You don't need a handful of tools just to do one simple task, just hack away at the zfs and zpool commands and everything gets done. No need for parted, mdadm, lvm or support of your needs compiled into the kernel. No matter on which operating system you are currently, you only need those two tools and they work the same everywhere.
Expanding a RAID vdev is not possible, instead the recommendation is to waste 50% of storage for a pool of mirrored devices to get the same protection as a RAID1 of two drives, which isn't much for some people.
For the avarage home user, such a scaling option is already no longer viable.
This is topped with ZFS' poor performance under low RAM conditions. FreeNAS barely managed 10MB/s when running with 2GB of RAM and an old Pentium 4 Dual Core. Not everyone can afford to throw hardware at the problem.
On the other hand, Linux performs under the same conditions with 8 times the speed for simply copying files and mdadm/lvm/unraid allow me to dynamically grow the pool of harddrive space a single drive at a time.
(plus mdadm/lvm is included in basically every larger distro by now, no need to install it or compile support into the kernel)
Or make a new vdev. It takes a bit of planning, but it's not some massive hurdle.
>This is topped with ZFS' poor performance under low RAM conditions. FreeNAS barely managed 10MB/s when running with 2GB of RAM and an old Pentium 4 Dual Core. Not everyone can afford to throw hardware at the problem.
FreeNAS is enterprise software, the default settings expect you to read and follow the system requirements. The developers aren't spending their time supporting it on less powerful systems, but ZFS (and likely FreeNAS) can run fine if you configure it for that hardware.
>Or make a new vdev. It takes a bit of planning, but it's not some massive hurdle.
I had a 3 disk, 6TB RAIDZ (3x 3TB drives). I bought a fourth 3TB disk when I started running low on space, not knowing that this limitation existed. I ended up using an intermediate, 8TB external drive to copy data off, re-create the RAIDZ with 4 disks, and then copy the data back.
This sucked. What should I have done instead?
RAIDZ expansion is coming though[0]. This is the right answer to your problem, but an answer at some point in the future doesn't help you with your problem today.
And, replacing disks one by one would require me to buy 3 disks instead of 1. I did that the last time I grew my array :-/
You should have read almost anything about ZFS before investing several hundred dollars and your data into it. It sucked for you, but seems like a reasonable ask.
I'm surprised you chose ZFS without knowing about it, it seems to be mentioned in every discussion about ZFS.
I was also making the point that "make a new vdev" doesn't solve that particular problem.
Overall, I'm quite happy with ZFS.
The hardest part on Linux if you're running a distro that doesn't officially support ZoL so you'd have to compile you're own kernel module. But even that isn't the problem it used be in the old days of Linux since we usually have bootstrapping scripts to handle the pain now (however if you can run FreeBSD then that's always a good choice as well. It's genuinely a really nice platform to use).
ZFS isn't quite easier to user for multiple disks or volumes, rather, there is a very tiny set of configurations that everyone recommends (RAID1 mirrors, leaving consumers unable to grow pools 1 disk at a time because ZFS is fundamentally unable to grow a RAID vdev). LVM and mdadm manage these jobs much better and are composable to your needs.
I've read people say that but I've never had any issues when I used to run ZFS on various systems with 2GB RAM. So I wonder how many of those people actually had practical experiences running ZFS on low memory systems and how much of it is people just looking at some of the recommended memory specifications (which are high) and start making assumptions about it's performance on low memory systems.
> RAID1 mirrors, leaving consumers unable to grow pools 1 disk at a time because ZFS is fundamentally unable to grow a RAID vdev
You actually can grow a ZFS vdev - exactly by replacing one disk at a time (albeit you don't get the gained capacity until all the disks in the vdev have been replaced).
What you cannot do is shrink it - but that's not a unique problem to ZFS and is a pretty uncommon use-case anyway. Also you cannot grow a ZFS "raidz" vdev by adding disks - but you cannot do that with any typical RAID5/6 array either. However if you want to add disks to a raidz then you can workaround the RAID5/6 striped parity problem by adding the additional disks into a new vdev and adding that new vdev into the existing ZFS volume (named "tank").
The kind of storage arrays that grow by adding solitary disks work quite different to your typical RAID. They'll often have designated parity drives instead. See solutions like "unraid".
> LVM and mdadm manage these jobs much better and are composable to your needs.
LVM and mdadm are not file systems though, so they only solve part of the problem. The fact ZFS handles volume management, software RAIDing, file system operations (and a bunch of other stuff that your typical fs doesn't yet support (eg snapshotting)) and does so from surprisingly intuitive command line tools is what edges ZFS for me in terms of usability.
Ultimately though, "easier" is such a broad term that we're essentially just arguing personal opinion. :)
I've ran it for real. Once a few more background services start running, ZFS bogs down and keels over. I've had to reboot the NAS dozens of times because the entire filesystem had become totally unresponsive.
>You actually can grow a ZFS vdev - exactly by replacing one disk at a time (albeit you don't get the gained capacity until all the disks in the vdev have been replaced).
Which is not an accaptable answer to the situation I presented, you still have to buy multiple disks to grow the pool, you can't do it one at a time.
>The kind of storage arrays that grow by adding solitary disks work quite different to your typical RAID. They'll often have designated parity drives instead. See solutions like "unraid".
Not only unraid, there are other systems that provide almost this. Syno and QNAP can both grow an array in storage capacity if you add a single new disk, even if it's larger and once you have added enough to provide parity, the additional capacity will become used too.
>surprisingly intuitive command line tools is what edges ZFS for me in terms of usability.
I found the commands anything but intuitive though I guess that's a difference of a few years of experience, I'd agree that would be personal opinion if it's easier.
Was that an off the shelf NAS? If so then I'm guessing it wasn't exactly high spec'ed even putting aside the RAM? eg did it have a 32bit CPU? I've heard that does a massive amount to dent ZFS's performance (I wouldn't know because I've only ever ran on it 64 bit systems).
As I said, I've not had any issues with raidz2 on 2GB RAM. I've even ran virtual machines on one OpenSolaris box with ZFS and 2GB RAM so these weren't always dedicated storage servers either.
edit: just read elsewhere you were running FreeNAS on a Pentium 4. Yeah that would do it. FreeNAS wasn't optimised for low end systems (in fact they outright tell you not to!) and the P4 is pretty low end CPU as well. I still think ZFS would have ran on that but you'd have been better off going with vanilla FreeBSD instead.
> Which is not an accaptable answer to the situation I presented, you still have to buy multiple disks to grow the pool, you can't do it one at a time.
...but you can't do that with most traditional RAIDs either and the few you can do it on require you to take the whole storage pool offline while you grow the pool then resize the file system. That process can literally take several days on some storage servers. So ZFS might require you to add vdevs to the pool (tank) but at least it can all be done on live systems without any downtime. That in itself is a massive boon for ease of use in my opinion.
To be honest, it's pretty rare you want to upgrade a storage array by just one disk anyway. At least outside of home users anyway. And if you are the type of person that would need to do that then you wouldn't go for a traditional raid5/6 style pool either. Thus the argument becomes a comparison of apples and oranges as you'e comparing completely different types of software RAIDing.
> Not only unraid
I didn't say "exclusively unraid". I said "solutions like unraid". I'm fully aware there are other solutions out there.
> there are other systems that provide almost this. Syno and QNAP can both grow an array in storage capacity if you add a single new disk, even if it's larger and once you have added enough to provide parity, the additional capacity will become used too.
When you said "there are other systems that provide almost this" and "once you have added enough [disks] to provide parity, the additional capacity will become used too." I really had to question what your actual point of this discussion was because just a paragraph earlier you were calling my points "unacceptable" because ZFS required you to add more than 1 extra disk to provide parity; and now you're saying your preferred solutions also does the same?
I hope I'm just being presumptuous here but it's starting to feel like you're just arguing for the sake of arguing. :-/
> I found the commands anything but intuitive though I guess that's a difference of a few years of experience
Maybe. I've been a Linux and UNIX sysadmin for a fair few years now so experience is not something I'm short of. :)
Honest question: what is your experience of ZFS outside of the troubles you had on your NAS? I could totally see why the problems you described would taint your opinion of ZFS but I definitely wouldn't say what you experienced is typical for ZFS.
In any case, I'm not going to argue that everyone should be running ZFS - it's just technology, you can run whatever you want to run.
OpenZFS is currently working on adding code to allow for vdev expansion[1]. I am a fan of ZFS as well, but I don't get not wanting to admit its limitations. Not being able to expand vdevs is a limitation and it's being addressed.
Or in other words: if growing the vdev is a hard requirement then a traditional raid might not be the best option regardless of ZFS. It’s about picking the right tool for the job rather than moaning that ZFS doesn’t do every job perfectly (which seems to be the argument the GP was making).
No it was a repurposed office computer running a 12 year old Pentium 4 Dual Core (about 2.8GHz per Core). 2GB of DDR2.
As you mention, FreeNAS isn't optimized for that but a plain BSD install with ZFS didn't fare much better either and required a rather inane amount of work compared to other solutions that worked OOTB.
>...but you can't do that with most traditional RAIDs either and the few you can do it on require you to take the whole storage pool offline while you grow the pool then resize the file system.
If you use mdadm and ext4 you can do it online.
>To be honest, it's pretty rare you want to upgrade a storage array by just one disk anyway. At least outside of home users anyway.
There is a lot of home users. People will use RAID5s and 6s at home, they'd rather have no-though-just-works solutions.
The home desktop is possibly the largest filesystem market next to "disposable virtual machine".
>and now you're saying your preferred solutions also does the same?
They don't behave the same. In these solutions if you add 2TB to a 4x1TB RAID5 array, you get 4TB of storage. If you add another 2TB you get 7TB of storage.
On ZFS with a RAIDZ1 you'd get 5TB of storage since you'd have to stand up two RAIDZ1 vdevs.
>what is your experience of ZFS outside of the troubles you had on your NAS?
I've been running a few hypervisors using ZFS as storage backend, though at the moment I'm migrating to a LVM thinpool based backend. I've had a few share of problems on that as well, notably ZFS at some point consume almost all of the 32GB of memory and running the OOM killer in the linux kernel until everything had crashed and the host required a hard reboot. Though this hasn't happened since.
That's pure conjecture. Plus I actually have ran ZFS on a Pentium 4 on FreeBSD. For a while that was my personal backup server.
> If you use mdadm and ext4 you can do it online.
Not according to any of the docs I've read in the past. Happy to be proven wrong if you have a citation though.
> There is a lot of home users. People will use RAID5s and 6s at home, they'd rather have no-though-just-works solutions.
> The home desktop is possibly the largest filesystem market next to "disposable virtual machine".
I don't disagree with that but how many of those users are running LVM + mdadm? And how many of those people are just using unraid (or similar purpose built solution for home users)? How many users are just using an off-the-shelf NAS like QNAP et al?
The issue I have with your complaints is that you're flip-flopping across a multitude of industries from desktop to enterprise, which seems grossly unfair for making a balanced opinion. I'm not going to deny that ZFS will lack some desirable features compared to some speciality tools but like I said before, it's about picking the right tool for the job rather than arguing that ZFS is garbage because it doesn't win every single argument in every specific speciality.
I mean you're not even willing to take on board any different configurations of ZFS nor alternative platforms it ships on - instead only basing your biases on one failed attempt at running FreeNAS (something which openly states they need beefy hardware to run) on equipment that is several generations old while then comparing that one specific ZFS instance against every alternative tool you've read about. That hardly seems fair, don't you think?
> They don't behave the same. In these solutions if you add 2TB to a 4x1TB RAID5 array, you get 4TB of storage. If you add another 2TB you get 7TB of storage.
No raid would give you 3TB of extra storage when you add a 2TB HDD. That would be breaking the laws of physics because you're now asking your server to create 1TB from thin air.
But to assume you meant 2TB of extra storage when you add a 2TB HDD: in your situation you don't get redundancy so your data becomes at risk if you have a failure. You can actually expand your ZFS tank the same way if you really wanted (but no-one who cares about their data would want to) by adding a HDD vdev (unraided) to an existing tank.
The issue is when you want to expand an existing raidz while retaining redundancy. To do that on any of the existing FOSS tools I've used, you'd have to either buy more than one HDD or do an offline restripe (as is the case with mdadm). I guess at least mdadm allows you to do it - even if it takes the storage pool offline for several days. Personally I've always opted to buy multiple drives and expand the pool (tank) by adding an additional raidz vdev because that not only expands the storage array but also then gives me another failure option (eg I can then lose 2 disks simultaneously without losing data). As you start to expand the number of disks you have, you equally increase the likelihood that more than 1 disk might fail. So having that extra redundancy disk is desirable to me.
> On ZFS with a RAIDZ1 you'd get 5TB of storage since you'd have to stand up two RAIDZ1 vdevs.
Actually no. You have to stand up two vdevs but a vdev can be a raidz, mirror, a solitary HDD or even a file on an existing file system. So you can just add an additional HDD to a ZFS tank but you wouldn't get redundancy on that additional HDD (which is why nobody suggests you should do it).
> I've been running a few hypervisors using ZFS as storage backend, though at the moment I'm migrating to a LVM thinpool based backend. I've had a few share of problems on that as well, notably ZFS at some point consume almost all of the 32GB of memory and running the OOM killer in the linux kernel until everything had crashed and the host required a hard reboot. Though this hasn't happened since.
Yeah, I don't much like how ZFS's cache isn't shared with the kernel as freeable RAM. I get why they did it that way and generally it's not an issue (aside monitoring / reporting tools) because the ZFS kernel module should deallocate memory itself when it detects system memory is getting low but clearly that didn't happen in your case (which is strange - sounds like you found a bug that was worth reporting). Frankly speaking, I'm not a massive fan of ZoL anyway. I've got it working before but honestly I much prefer running ZFS on FreeBSD or Solaris / OpenSolaris,
With regards to your particular use case: I wouldn't have recommended running ZFS for your hypervisor backend anyway. You don't really want CoW for write-heavy processes - particularly ones that have their own journals and/or snapshotting systems such as databases and hypervisors would. Like I said before, it's about using the right tool for the job and XFS would be my preferred Linux file system in those particular instances.
https://raid.wiki.kernel.org/index.php/Growing
ext4 online resize works by invoking resize2fs.
>I don't disagree with that but how many of those users are running LVM + mdadm? And how many of those people are just using unraid (or similar purpose built solution for home users)? How many users are just using an off-the-shelf NAS like QNAP et al?
QNAP and Synology don't do magic. They deploy mdadm, LVM and ext4 or btrfs (though btrfs only in single drive mode, they use LVM and mdadm to span the volume for btrfs).
>That hardly seems fair, don't you think?
It's the real world experience I had with the software and I think it's totally fair to point out and complain about the shortcomings.
>No raid would give you 3TB of extra storage when you add a 2TB HDD. That would be breaking the laws of physics because you're now asking your server to create 1TB from thin air.
Reread what I wrote, I mentioned adding 2x2TB not 1x2TB.
> in your situation you don't get redundancy so your data becomes at risk if you have a failure
Still having redundancy is exactly the point. The resulting array in the example would span 6x1TB and 2x1TB in RAID5, giving me a single drive for parity in each span and resulting in an effective RAID5 for the entire RAID. I can loose any drive in that 4x1+2x2TB array without loosing data.
And it does that online without having to reboot or even halt any ongoing file operations. The volume will simply increase in size once it has finished formatting the new drive and integrated it into the array.
>So you can just add an additional HDD to a ZFS tank but you wouldn't get redundancy on that additional HDD (which is why nobody suggests you should do it).
Again, what Syno offers (and what can be replicated on any Linux system) allows you to add a single drive to expand the storage without loosing parity and more drives to span any drive increases with parity. This allows adding a single drive at a time and at worst you loose any additional capacity without parity yet (in the 1TB/2TB example, that would be loosing 1TB of storage until the second drive is added)
That's resizing the filesystem not the pool (eg `mdadm --grow`). You're also missing the point about what I said. To run those commands you must first `umount` to pool. Hence it being an offline job.
> QNAP and Synology don't do magic. They deploy mdadm, LVM and ext4 or btrfs (though btrfs only in single drive mode, they use LVM and mdadm to span the volume for btrfs).
That's completely irrelevant to the point I was making. How many home users do you know who configure their own home raids using LVM and mdadm?
> Reread what I wrote, I mentioned adding 2x2TB not 1x2TB.
You actually said neither. You said, and I quote, "2TB" (ie without any accumulation prefix).
> Still having redundancy is exactly the point. The resulting array in the example would span 6x1TB and 2x1TB in RAID5, giving me a single drive for parity in each span and resulting in an effective RAID5 for the entire RAID. I can loose any drive in that 4x1+2x2TB array without loosing data
You can actually do that in ZFS. In fact what you've just described were is the exact solution that you called "unacceptable" in ZFS!
> And it does that online without having to reboot or even halt any ongoing file operations. The volume will simply increase in size once it has finished formatting the new drive and integrated it into the array.
Indeed. Lots of commercial tools do support that. I thought we were comparing ext4, LVM and mdadm to ZFS though?
I'm guessing you just glossed other my post where I said you were flip-flopping all over the place? It's easy to pick any specific tool ("tool a")and compare it to a thousand other tools and argue that "tool a" is crap because it doesn't work as well as tool b, c and d in very specific edge cases but all you're doing is trying to win an argument via death from a thousand papercuts rather than forming a sane basis for analysis.
To be honest I think it's probably best we give up on this discourse because your points are all over the place. This feels very much like you've got an axe to grind and no practical experience to back up your prejudices.
No you don't need to unmount the pool under mdadm. You can do a live resize of the array. See the link I posted.
>How many home users do you know who configure their own home raids using LVM and mdadm?
Not many, hence me mentioning Syno and QNAP.
>You said, and I quote, "2TB" (ie without any accumulation prefix).
I said, quote, "if you add 2TB to a 4x1TB RAID5 array, you get 4TB of storage. If you add another 2TB you get 7TB of storage."
>In fact what you've just described were is the exact solution that you called "unacceptable" in ZFS!
Again, this is not what ZFS does, the mdadm solution stripes multiple RAID arrays over the same drive, which to my recollection is not a recommended configuration in ZFS by a long shot.
>Lots of commercial tools do support that. I thought we were comparing ext4, LVM and mdadm to ZFS though?
The commercial tools I mentioned previously use ext4, LVM and mdadm native functionality.
By default, sure. There is a FreeBSD (I think) guide to tuning ZFS that lets you tune it for low memory. It's a long time since I've done this so I can't remember which settings, but I ran an old machine tuned for its low max memory, with 4 drives in RaidZ, and it was just fine.
The FreeNAS forum doesn't help much with this as the default opinion there seems to be 16GB ECC. :)
As you mentioned, the bitrot. Also, my understanding is that a COW filesystem can be more reliable than a traditional journaling filesystem in a system crash since nothing is overwritten. Plus there is compression and all of the features provided by a COW filesystem.
The risk of bitrot becomes interesting when you talk about long timeframes like a decade.
Considering the capacity and price of a 2TB disk, compression isn't the immediate need of the consumer either, unless they have a NAS, where Syno and QNAP already provide integrated solutions with btrfs and ext4 both.
The modern ext4 journaling is also very battle tested, it's a very simple and resilient physical journal, data loss in ext4 happens largely when the disk is already failing or has been cheap enough to lie about the state of data being written or not.
For the average consumer that sums up to "it doesn't matter enough". Something like bcachefs could change the game by being btrfs+ZFS but without the dataloss risk.
Of course I’ve been using btrfs RAID1, so perhaps bit rot has been silently corrected.
> The risk of bitrot becomes interesting when you talk about long timeframes like a decade.
I think you are making a lot of assumptions here about frequency of bitrot, especially in the face of failing hardware. Also, consumer desktops are not the only use. There are bunch usages where the risk of bitrot is much more concerning. Or were you intending your original post to be only about consumer usage?
>Considering the capacity and price of a 2TB disk, compression isn't the immediate need of the consumer either, unless they have a NAS, where Syno and QNAP already provide integrated solutions with btrfs and ext4 both.
This kind of feels like you are saying that it isn't needed because it is already being used when it is needed. Am I misunderstanding? Consumers may want compression for smaller hard drives they already own in order to get more out of them. Also, compression can speed up disk IO in some workloads.
> The modern ext4 journaling is also very battle tested, it's a very simple and resilient physical journal, data loss in ext4 happens largely when the disk is already failing or has been cheap enough to lie about the state of data being written or not.
The issue with ext4 resilience isn't about quality of the implementation. It is a fundamental limitation of no COW filesystems. Since things are overwritten, if the system crashes mid write you can end up with partial data. COW filesystems always give you the old data or the new data. This can be worked around but that requires that all of your applications do this workaround.
> For the average consumer that sums up to "it doesn't matter enough". Something like bcachefs could change the game by being btrfs+ZFS but without the dataloss risk.
What makes you think bcachefs is going to be any better? A lot of functionality still has to be implemented. What makes you think that the implementation of this functionality will be any less error prone?
You also haven't addressed the large pile of new features provided by BTRFS/ZFS. Snapshotting alone can make it worth the price of admission for consumers. With snapshotting, you can set up a system where rollbacks after any upgrade can occur seamlessly.
You do know how journalling works, yes? Unless the drive lies about the state of data being written, in which case not even COW will save you, a journal is just as good as COW. Exceptions being coding errors but those can occur on COW with the same devastating result.
Maybe you don't know as much about filesystems as you want to appear to?
Yes, I'm aware of how journaling works. The problem with your description is that it requires both data and metadata journaling. This is rarely the case because data journaling requires that you write all of your data to the disk twice (which also requires additional seeks as well). So, nearly all in-place filesystems default to only metadata journaling for performance reasons (this is the case for ext4). This means that, in practice, if you get a system crash mid file write, you will end up with a partially updated file. On a COW filesystem you are always guaranteed the old file or the new file.
> Maybe you don't know as much about filesystems as you want to appear to?
That is possible, although I'm not sure how much I want to appear to know :P. More seriously, this is extremely condescending. Why is this necessary? If you think I'm wrong, just tell me I'm wrong. There is a pretty decent chance I'm wrong about most things. So, condescension shouldn't be required.
Compared to just running gdisk and using raw partitions - sure, but I’ve been using LVM for over a decade now.
Even Synology and QNAP now have plug-and-play solutions to expand a RAID one disk at a time. You can even start plugging in higher capacity disks and once you have enough extra parity, the higher capacity becomes used magically.
No need for wasteful RAID10/50/60 setups. Just plug in more drives.
This also affects the RAID1 setup, any storage expansion requires 2 disk to pull up a new RAID1 into the pool.
I don't know where you read that, but don't be fooled. That functionality has been "in the works" for quite a long time. But it hasn't yet appeared.
In the past it has been said that something called "block pointer rewrite" was a prerequisite. Just google for that.
Here's some 10 year old information about it. At that time it was already a "frustratingly frequent request." I stopped searching when I found something 10 years old, but I'm sure you with some effort you could go back much further than that.
Those RAID set ups are only "wasteful" if you don't need that level of redundancy (ie the number of disks that can fail at the same time). Most home users only really need 1 redundancy disk (actually most home users don't even need that) but a production database server is another matter entirely.
For clarity, the reason we have redundancy drives isn't to mitigate the need for backups; it's the mitigate the likelihood of downtime while you rebuild your storage array from a backup. Which is why most home users don't even need 1 redundancy disk - they just need to be disciplined in backing important stuff up.
Scenario 1: Replace smaller disc with bigger disc. Add new disc as mirror to vdev. Wait for it to resilver. Remove original disc from vdev. Pool will autoexpand to new disc size or can be manually expanded otherwise.
Scenario 2: Expand virtual storage. Use LVM lvresize, or expand VM virtual disc. Pool will autoexpand to new volume size; may need manual triggering if the system doesn't pick up the new device size automatically. Or you can do the same as for physical discs. Add new virtual storage as mirror, resilver, then drop the old storage.