Hacker News Reader

Comment by NovaS1X | Hacker News Reader

What are the few warts you encounter? I'm a Sysadmin/Engineer at a VFX company. We have hundreds of machines, render, and servers all running Linux but we're typically on ext4 with the notable exception of our Isilon cluster.

What does your environment look like?

View on HN

As a few others mentioned, the 80% issue is one. We try and keep all machines below that so performance doesn't degrade. With the number of drives we have (200+ PB total space) we also run into issues where there are "holes" in the ZFS array after replacing a disk. We've always had enough redundancy work around any issues we find. Overall, I cannot speak highly enough of ZFS!

One that I've heard often repeated (paging user rsync) is that you should never fill your filesystems past 80% (or so). Things get slow otherwise.

ZFS certainly slows down as it fills up, but only enough to affect high-IO, write-intensive loads like databases. A light-weight application like a home or office file server can still saturate gigabit ethernet even while 99% full.

This is also true of other filesystems however. I don't have experience with high IO on ZFS at those rumored 80%+ capacities however once a core filer fills up to 90% you start seeing the tickets rolling in and application performance tank. 80% if usually fine however. I wonder to what degree the performance degradation is on OpenZFS.

The problem is much more severe on ZFS and other CoW filesystems, hence the common warning. Filling an ext4 filesystem up to 80%+ and then removing files might increase fragmentation a bit but it'll not have that much effect on performance. On ZFS, filling the fileystem up once can kill performance permanently on that pool.

> you should never fill your filesystems past 80% (or so). Things get slow otherwise.

That's a ZFS-in-general complaint, not a ZFS-on-Linux complaint.

However, it's worth mentioning most (all?) filesystems drop sharply in performance as it nears capacity.

"One that I've heard often repeated (paging user rsync) is that you should never fill your filesystems past 80% (or so). Things get slow otherwise."

Hi. I've said a lot about this in HN comments, so I encourage you, future reader, to search for that. It is indeed the case.

Recently it has been suggested that the presence of a fast write cache (a SLOG, in ZFS parlance) minimizes this problem and allows you to run up to, and around, 90% without breaking the filesystem.

We haven't tested this, intentionally or otherwise, but it sort of makes sense ...

Make no mistake, however - if you fill up a ZFS filesystem and run it for a while in that state, it will be permanently broken, at least in terms of performance.

A ZFS defrag utility would solve this problem, or at least provide a way out if you fall into this trap, but it has been related to me that ZFS defrag would be extremely complicated to implement.

Yes, we do indeed see this on other filesystems ... even a UFS2 filesystem with NO snapshots enabled can be effectively "broken" is you set minfree down to 0%, fill it all the way up, and then run it like that for a while. Freeing up the space and resetting minfree back to 6 or 8%, etc., doesn't fix it.

Yes, because after 80% ZFS will consume CPU(s) in an attempt to find enough contiguous space for the new branch of blocks in order to prevent fragmentation. The fix is to either add more vdevs or to change the kernel tunable which controls the free contiguous space allocation algorithm. The latter approach is very likely to cause fragmentation where there would otherwise be none. As I don't remember the name of the tunable, you can search my comments and in one of them on the subject there will be a link to the procedure. However, I strongly recommend adding more logical units or physical disks as vdevs to the zpool instead.

Reply on news.ycombinator.com