Comment by aroch | Hacker News Reader

Comment by aroch | Hacker News Reader

Parent Full thread

ZFS pool IO performance drops like a rock off a very tall cliff once the pool is in the 80%+ full range. So for every TB of usable pool space, you need to over-provision by 200-300GB. This is with current gene enterprise SSDs mind you. Even worse things happen when using spinning rust -- like ZFS dropping drives from the pool because IO has degraded so much it thinks theres a bad drive.

It turns out that some raid controllers (Including the very popular LSI variety) + certain kernel versions result in pools and drives being dropped from the array for no discernible reason. Nothing [useful] to logs, just reboot and pray

Non-ECC RAM is just asking for trouble in any circumstance where you care about both data integrity and IOPS. Bitflips can and will get you, now they won't wind up hosing your entire pool (most likely) like some apocryphal stories suggest. But, you'll get bad data replicated and the possibility of borking some other sector during the 'recovery' process from the bitflip.

When you're in a large enough pool (PB-scale) this becomes even more painful...

100% true about the performance cliff, though at Delphix we worked a bunch to raise performance before the cliff and to push out that cliff. All COW filesystems filesystems (and I'll throw SSD FTLs into there as well) require over-provisioning for consistent performance.

And it's also a fair point that for enterprise storage running at 80%-90% of capacity is a reasonable restriction whereas the drive on my laptop is always basically %99 full. It will be interesting to see how APFS addresses this since I'd guess that most of their users look more like my laptop than my file server.

We always over-provision any way, but with ZFS the overage is a requirement for any reasonable performance. With XFS, you can get away with 90-95% full; yes, there's degraded performance but nothing like the 10 IOPS you get from ZFS.

Don't get me wrong I like ZFS/btrfs; I adore snapshot send/receive. It at times though really handicaps itself.

I did a brief write-up about this effect on the ZoL issue tracker:

https://github.com/zfsonlinux/zfs/issues/4785#issuecomment-2...

In short, if you want more consistent performance as the pool fills, disable metaslab_lba_weighting_enabled, but be prepared to lose some sequential performance when the pool is empty.

It's my understanding that you shouldn't use a RAID controller with ZFS...

LSI cards can be run in both RAID mode and passthrough / standard SAS/SATA controller mode for XFS/ZFS/btrfs -- which works wonderfully most of the time and cuts down on hw heterogeneity. You will have to use some disk controller or another, your motherboard is not going to have 12/24/36/48/80 SATA ports

Nor a battery-backed write cache.

I believe adding a ZIL device alleviates the low free space performance issue.

Adding a ZIL device is equivalent to over-provisioning, but with performance impacts in high performance situations

From what I understand a ZIL helps with workloads which need to fsync a lot (say databases) and also NFS (due to COMMIT commands issued by clients).

Reply on news.ycombinator.com