Ask HN: Btrfs or ZFS on Linux in production? | Hacker News Reader

Ask HN: Btrfs or ZFS on Linux in production?

9 pointssmilliken13 comments

ZFS but, tune it properly by limiting the size of ARC and buy some SSDs to use as both log devices and cache devices.

Also, do not buy any controller cards that will get in the way of ZFS talking directly to the disk (such as Dell PERC cards), where you see a "virtual" disk instead of the real one, it will affect performance.

Thank you. I have an old Optiplex as a number crunching toy and had just decided to use the two currently unused SAS drives to run a first ZFS test. You just saved me a lot of frustration.

Regarding the ARC, is zfs not as efficient about evicting the cache as, say, ext4? With ext4 I've never had to worry about it consuming as much memory as it pleases.

Thanks for the tips!

My layman's understanding: under Linux the ARC is not integrated into the usual Linux caches which are unified and which the kernel understands. So basically the ZFS ARC is a separate chunk of RAM, which can grow and grow (thus you need to limit it) and it won't be evicted as perfectly as the kernel would evict stuff cached on xfs or ext4 filesystem.

Ah, that makes sense. Thanks for the explanation.

They are both check-summed and copy-on-write.

This is great for consistency and storage, but the first part somewhat sucks if you've built it into your layer as well (like hadoop fs). And it makes writes & reads a tiny bit slower than it should be.

They have fat xattr inodes, which is great if you use it - but useless if something like SELinux means nothing to you.

The point where they truly excel is in a multi-disk scenario.

And in that land, the zpool array implementation makes it awesome to introduce an SSD+HD mixture with nearly the best of both systems.

BTRFS doesn't have that yet (well, bcache on the lower layer, perhaps).

But as someone who's fired up debugfs a bit too many times, if you really don't know which one you need, pick the one that ships with your distro!

> And in that land, the zpool array implementation makes it awesome to introduce an SSD+HD mixture with nearly the best of both systems.

With an array of half SSDs and half HDs, can I configure ZFS so that I never hit the IOPS bottleneck of the HDs? How do you recommend configuring the array?

I have never used a setup like that - all I've done is throw an SSD in front of 3 disks.

That said, the zfs log and zfs cache command supports mirror disks for log and allows more than 1 cache per pool.

Using more than 1 SSD as a cache (and of course, using mirrored logs for safety) would be recommended if you had a production box with this.

To repeat, I have never tried ZFS-on-Linux anywhere near a production setup - but if I had to, I would use multiple SSDs just in case one locks up its firmware.

Choose extfs every time. btrfs is bloated.

x220:~/tmp/linux-3.13.2/fs/ext4$ ohcount . Examining 39 file(s)

							  Ohloh Line Count Summary

	Language          Files       Code    Comment  Comment %      Blank      Total
	----------------  -----  ---------  ---------  ---------  ---------  ---------
	c                    37      31707       8586      21.3%       5038      45331
	make                  1          8          3      27.3%          3         14
	----------------  -----  ---------  ---------  ---------  ---------  ---------
	Total                38      31715       8589      21.3%       5041      45345
	x220:~/tmp/linux-3.13.2/fs/ext4$ cd ..
	x220:~/tmp/linux-3.13.2/fs$ cd btrfs/
	x220:~/tmp/linux-3.13.2/fs/btrfs$ ohcount .
	Examining 81 file(s)

							  Ohloh Line Count Summary

	Language          Files       Code    Comment  Comment %      Blank      Total
	----------------  -----  ---------  ---------  ---------  ---------  ---------
	c                    79      77101      13378      14.8%      13025     103504
	make                  1         15          0       0.0%          4         19
	----------------  -----  ---------  ---------  ---------  ---------  ---------
	Total                80      77116      13378      14.8%      13029     103523

I'd be surprised if btrfs was smaller. Its feature scope is significantly wider.

The btrfs performance isn't at par yet either, but I'm particularly interested in filesystem-level compression which isn't supported in ext4.

Both have good features, both will be useful and interesting in a few years.

Right now I'd not trust either of them, in production use. Even if you never spent 60 minutes restoring backups both filesystems are too new for me to trust, with my conservative tendencies.

My understanding is that btrfs is the likely successor to ext4, but that zfs on linux is further along and more featureful.

Are there any data integrity, performance, or other issues to beware of?

I would go with ZFS, at least for now. BtrFS is getting there but i wouldn't use it production just yet,

Reply on news.ycombinator.com