Comment by aidenn0 | Hacker News Reader

Comment by aidenn0 | Hacker News Reader

Parent Full thread

I've used both ZFS on both BSD and OpenIndiana, and I've used btrfs on linux. as recently as 4.0.5

As of 4.0.5 btrfs was IMO completely unusable as a daily file system. Some examples of issues I ran into:

1) System became unbootable with the version of btrfs I had installed and I had to use either an older or newer kernel to recover

2) I have a periodic backup of my mailbox that runs, and when it runs my system becomes completely unusable until it completes. The same script running on zfs on bsd and with ext4 or reiser3 on linux would show I/O slowdowns, but I could still use my machine.

3) In general I would run into other minor issues and the consensus in #btrfs was that since my kernel was more than 3 months old, it was probably fixed in the latest version, and why would somebody using an experimental filesystem not be tracking mainline more closely?

[edit]

To be fair, here's some issues with ZFS:

1) Do not ever accidentally enable dedupe on commodity hardware; it will slowly consume all your RAM if you aren't on a sun server (where 64GB of RAM is a resource constrained environment), and there aren't effective ways to undo dedupe, other than copying all the data onto a different pool.

2) You can't shrink a pool. Hugely annoying, apparently non-trivial to solve.

"To be fair, here's some issues with ZFS:"

Let me add:

3) Do not allow a pool to exceed 90% capacity ... and probably don't let it exceed 85%.

ZFS does not have a defrag utility and it badly needs one. You can permanently wreck zpool performance by running it up past 90% capacity - even if you later reduce capacity back down to 75-80%. You can sort of fix it if you add additional top level vdevs to the zpool, thus farming out some IO to the new set of disks, but it's still going to be performance constrained forever. The only solution is to create a new pool and export the data to it.

This is unacceptable, by the way.

It is not at all reasonable to require a filesystem to stay below 80% capacity (our target "full" number at rsync.net) nor is it acceptable that hitting 90% is a (performance) death sentence.

When you consider that you might have already sacrificed 25% of your physical disk just for the raidz2/raidz3 overhead, being constrained to 80% means you're only using 60% or so of your physical disks that you bought.

ZFS needs defrag. Badly.

The Reiser guys were talking about this too. I think they needed defrag and were trying to get DARPA money to implement it, but it never happened.

For ZFS to get defrag, someone with enough ability and knowledge would need to bite the bullet and go through the fairly massive and difficult task of adding in block pointer rewrite, which doesn't seem to be something anyone has been willing to do, and I've seen a lot of concerns about the actual feasibility of it from some very smart people that are knowledgeable with the codebase.

Which also incidentally would solve the pool shrinking problem as well.

I wonder how much money you'd need to pay a very talented engineer to do the work. This is yet another thing I didn't read about before building a home NAS on ZFS.

Maybe off topic, but Alex Reece from Delphix had a blog post about a year ago about the ability to remove a disk from a pool.

http://blog.delphix.com/alex/2015/01/15/openzfs-device-remov...

This would imply some kind of rebalancing, if I understand correctly. Which maybe could be considered a form of indirect defragmentation.

That's a good point about the 90% capacity.

Also the same "technically challenging" problem that needs to be solved for shrinking pools would also make defrag possible.

Not all of the performance issues encountered when a pool exceeds 90% capacity are related to fragmentation.

Some implementers have been able to significantly reduce the pain of those situations.

The main factor seems to be the best-fit allocator, which tends to go into action sooner from metaslabs crossing the 94% threshold earlier than the pool itself and still be selected due to LBA weighting, which is a trick to increase throughput on rotational media. This ought to help prevent best-fit allocation from occurring earlier than necessary on SSDs:

https://github.com/zfsonlinux/zfs/commit/fb40095f5f0853946f8...

That said, Delphix made changes to consider on fragmentation rather than space usage when the histogram is in use, so the performance before best-fit behavior goes into effect is better than outright selection of metaslabs by free space.

I had this happen to me... I accidentally allowed a pool to fill up to capacity and couldn't do anything with it because deleting files wouldn't free space due to snapshots and the commands to delete snapshots wouldn't work.

Then added a disk to it to try to recover. That worked, but only after adding a disk did I realize that I couldn't shrink the pool down again. I ended up moving the whole thing off to a new disk cluster and back again. Really painful.

If gang blocks are generated, you get more I/Os than necessary and an extra level of indirection, but the ZFS code base tries very hard to avoid that situation by switching to the allocator behavior at the metaslab level to best fit, such that most data written to the pool would not have gang blocks used at all. This is understandably terrible for IOPS, but it is likely the source of the performance degradation that you saw. You can probe for the zio_gang_* functions during a scrub to see if any gang blocks exist. On my pool that has exceeded 90% on multiple ocasioks, there are zero gang blocks and consequently, no permanent degradation from them. The only other problem that you might have (which tends to be caused by the order of writes rather than the fullness of the pool) is lower sequential performance from nonlinear block placement (one ZoL user measured this as cutting sequential reads in half on a pool filled with files made by bit torrent), but that is a much less severe problem, especially on solid state drives. If you want to fix placement, you can do a file copy or send/recv. The new locations should have blocks picked in sequence whenever possible.

I've been using btrfs since 2 years (Fedora) and the only problem I've ever run into was "no space left on device", solvable with a rebalance. btrfs also survived many hard poweroff.

I tried to use the mirror functionality when it was new. I tested booting with one disk missing. Errors all the way. I went into IRC and chatted with btrfs folk about the "bug". Their response?

"Booting without all members of your mirror is unsupported."

It's certainly supported now. mount -o degraded. It ought to be the default, though.

if it's not default yet it's a serious bug. a server should still come up completely with one disk in a mirror.

I'll pitch in with a neutral position. I've been using btrfs for four years, and in that time I've had unrecoverable fs corruption probably three times. This is on Arch, on bleeding edge kernels, where new releases are prone to regressions that break the filesystem.

But there was a tangible progression from instability towards increasing stability. I haven't had one lick of issue with btrfs in about a dozen kernel releases. I'm close to saying I'd trust in a production environment, since I use it everywhere else as a daily driver, I would just use an LTS release version just to be safe.

It is not all sunshine and roses, though. While Facebook employs several major btrfs developers, a lot of features that have been talked about for years still have not seen the light of day or any development whatsoever. lz4 compression, better checksum algorithms, per-subvolume encryption, online filesystem checking, and the Raid 5/6 support is still kind of garbage a year later. I worry that btrfs is suffering from a lack of interest in actually making the last legitimate pushes it needs, and code audits, and integration testing, to make it truly trustworthy.

But at the end of the day checksum integrity and COW are basically a game changer for me in terms of data integrity.

Reply on news.ycombinator.com