Hacker News Reader

Comment by aruggirello | Hacker News Reader

But is it, performance-wise? How will ZFS compare to btrfs?

I would expect ZFS to outperform btrfs because of ARC and ZIL. ARC allows ZFS to increase the cache hit rate. ZIL permits ZFS to update its on-disk trees asynchronously. I am not aware of any analog to this in btrfs. Desktop users typically report that their systems become more responsive after switching to ZFS.

The only case where I suspect btrfs might significantly outperform ZFS involves getdents() when the caches is cold. This is because ZoL does not presently do readahead on directory lookups. This has a noticeable impact on the performance of `ls` on the first access of a directory on mechanical storage, but it is not a problem for subsequent reads or on solid state storage. Prefetch support in getdents() will likely be implemented in either the next release or the one that follows.

At this point in time, they still can't be compared, because btrfs is still not [considered] production-ready, and subject to significant changes (also performance-related).

This is especially important because on use cases where performance differences are significant, that is, not on general desktop usage, the maturity of the FS is funamental, and btrfs is discouraged right now.

There are many situations where performance is important, and durability is... not. For example, ephemeral CoreOS cloud instances running Docker containers: they use lots of copy-on-write layers, but they don't actually need to persist any state across reboots (the layers may as well be stored in volatile memory.) Btrfs is perfectly "production-ready" for this particular use-case, so a [current] performance comparison would be pretty useful.

I am working on ZFS support for CoreOS. A snapshot of my WIP proof of concept was posted by my employer yesterday:

https://github.com/ClusterHQ/flocker/blob/zfs-on-coreos-tuto...

CoreOS uses btrfs as its rootfs. I imagine that the CoreOS developers managed to avoid issues like ENOSPC by virtue of not writing to their rootfs very much. I did not have that luxury since I compiled a Gentoo GNU userland on top of it during the course of development. I encountered numerous ENOSPC errors on btrfs when developing the ZFS port to CoreOS and even hit ENOSPC errors when trying to correct the btrfs ENOSPC errors with `btrfs balance /`. I would not consider btrfs ready for production use, but your mileage will vary.

When were you hitting these ENOSPC errors, and what kernel were you using when you hit them?

  $ df -h .
  Filesystem      Size  Used Avail Use% Mounted on
  /dev/dm-1        42G   41G  280M 100% /home
  $ btrfs fi df .
  Data, single: total=40.48GiB, used=40.21GiB
  System, single: total=4.00MiB, used=12.00KiB
  Metadata, single: total=1.01GiB, used=649.18MiB
  $ uname -r
  3.15.2-hardened

As you can see, I've been working with a pretty much full btrfs volume. It used to be TERRIBLE to deal with btrfs in such a situation, but I haven't had an ENOSPC issue in ages.

The Gentoo Prefix boootstrap on a developer image:

https://www.gentoo.org/proj/en/gentoo-alt/prefix/bootstrap.x...

CoreOS uses Linux 3.15.y.

You may be interested in Phoronix[1] linux review site. When I used to follow it through RSS more diligently, I routinely saw filesystem benchmark shootouts such as this[2]. I'm not sure if there's a more recent one than that, but there are more recent benchmarks of Btrfs and ZFS against prior versions of themselves so you can see improvements (or regressions).

1: http://www.phoronix.com

2: http://www.phoronix.com/scan.php?page=article&item=linux_313...

I would be really really careful with Phoronix benchmarking. If I recall correctly, I've seen them 'benchmark' and compare a FreeBSD system with a recent (at the time) Linux release. Issue was, they weren't even running the tests on the same hardware. <sarcasm> Really?! you saw performance differences between and i5 and i7?! Each running a different OS?! Get outa here!! </sarcasm>

I'd be interested in seeing that. It's been a while since I really followed the site, but I seem to recall he was at least forward about what he was testing and why in the text around the benchmarks, I will admit to him seeming to be overly eager to get articles out, sometimes of ambiguous usefulness, probably due to his advertising model (at least at the time, when I was there earlier it seemed far less riddled with advertising).

Edit: Nevermind about the advertising, it's still intrusive, I'm just running adblock now.

Phoronix does try to be honest about its methodology, but the person running benchmarks gives no thought to what his results mean, if anything. It shows in the distinct lack of documentation on why a given benchmark matters.

As for Phoronix's ZFS benchmarks, the test hardware used drives that misreported themselves as having 512-byte sectors, which handicapped ZFS performance. Phoronix rejected all suggestions that it correct for this as end-users had been doing. Phoronix refused to meet half way by posting two results (one with proper configuration and one without), and also refused the suggestion that it to mention the existence of that problem in its test hardware. I eventually wrote code to identify drives known to misreport their sector sizes so that ZFS will automatically use the correct settings on them. That lead to the Phoronix August 2013 benchmarks showing a remarkable improvement in ZFS performance in FIO. It was so great that it sparked a discussion among the btrfs developers:

http://comments.gmane.org/gmane.comp.file-systems.btrfs/2754...

Later that month, I publicly criticized Phoronix for posting misleading benchmarks:

http://phoronix.com/forums/showthread.php?83731-ZFSOnLinux-0...

Phoronix has not posted ZFS benchmarks since that time.

That's unfortunate, but doesn't surprise me all that much. Not because I think/thought of Larabel as likely to do something like that, but because I believe it's all too easy for organizations that are a single individual or organizations where a disproportionate amount of operation and decision making is really a single individual to make calls based on time and ability, and then fall back to defending that decision ever more vehemently long past the point where it should have been reassessed. Honest trusted feedback is priceless.

Anecdotally I see better performance using ZoL than I did when I was running the same zpool and hardware on FreeBSD.

This would suggest that the block device layer under ZFS and/or the VFS above it have room for improvement on FreeBSD.

ZFS supports better compression algorithms. Those can even speed up slow disks.

btrfs is almost painfully slow at times, so hopefully ZFS is a lot better.

Even on Solaris, ZFS doesn't have a great reputation for performance.

Can you substantiate that? ZFS on Solaris produced excellent numbers back in 2008:

https://blogs.oracle.com/brendan/entry/a_quarter_million_nfs...

More recent derivatives of OpenSolaris were reported to do 1.6 million IOPS last year:

http://www.high-availability.com/high-availability-zfs-partn...

Your mileage will vary, but ZFS has always had strong performance.

One of those is an 8 year old benchmark, the other used a 9.6TB flash array so hardly a realistic storage platform. I'm not saying that ZFS performance is awful, just that other features like data protection come first. If you're hoping for some salvation from btrfs you might be waiting a while. Of course they're both developed by the same company now!

Here's a benchmark from 2013 with most results showing that it has worse performance on Linux than XFS and EXT4:

http://www.phoronix.com/scan.php?page=article&item=zfs_linux...

Oracle has nothing to do with the Open ZFS community these days.

As for the benchmarks you cite, there are several key problems:

1. You say that they show ZFS having worse performance than XFS and ext4, but XFS is not in those benchmarks and the FIO tester shows ZFS as outperforming its competition by a significant margin.

2. They use a single disk. No server does this and while desktops and laptops do this, it is not clear how the benchmarks are of any relevance there. Additionally, LZ4 compression is not in use, when practically everyone deploying ZFSOnLinux would configure it to use LZ4.

3. ext4 manages to perform better than the theoretical limit of the SATA II interface and there is no discussion as to why.

4. ZFSOnLinux 0.6.2 is an old release. The most recent 0.6.3 release includes a new IO elevator and other improvements that enable ZFSOnLinux 0.6.3 to outperform its precedessor by a significant margin in many workloads.

Would you post something constructive that you actually did yourself? I am beginning to think that you have never even used ZFS.

Compared to what exactly? It's consistently competitive against other similar FSes (XFS, JFS, etc..) When using caching, it's supposedly the best for database storage. And, feature wise, it beats everything else out there.

As with most things, this isn't a binary decision, but typically ZFS is a good (if not the best) solution for most storage arrays.

Most people I know that are looking for performance on Linux stick with XFS, but then again people that are serious about Linux aren't running ZFS anyway. And of course there aren't any benchmarks for XFS vs ZFS on Solaris. Where I see consistent problems with ZFS performance is with metadata operations. Sure it has great features for managing disks and protecting data, but those are the primary feature set and performance has always taken a back seat.

Would you elaborate on what metadata operations are slow? If you had problems, I would like to know so that I could look into fixing them. Otherwise, I do not see anything actionable here. Your criticism so far contains zero verifiable claims.

As for XFS, it is a single block device filesystem that relies on external shims to scale to multiple disks. ZFS can outscale it when various shims are put into place to allow multiple disks to be used. One user on freenode had difficulty getting good sequential performance from XFS + LVM + MD RAID 5 on 4 disks. He reported that he could not get better than 44MB/sec writes while ZFS managed 210MB/sec. I had a similar problem in 2011 with ext4 + LVM + MD RAID 6 on 6 disks. In that case, I could only manage 20MB/sec. It is why I am a contributor to the ZFSOnLinux project today. To make these anecdotes constructive, it would be nice if we had documentation on how to configure XFS + LVM + MD RAID 5/6 in a way that sequential performance does not suffer. In my case, my performance issue involved KVM/Xen guests. I never confirmed whether that user ran his tests on bare metal, but I suspect that he did.

XFS' inability to scale past a single block device is not its only issue. Until a disk format change occurs that will add checksums, it has none to protect itself against corruption. When it gains them, it will not be able to do anything about that corruption when it detects it (aside from keeping the kernel from panicing) and it does nothing to protect your data.

That said, ZFS has always focused on obtaining strong performance. This is why it has innovations like ZIL, ARC and L2ARC. The only instances where ZFS purposefully sacrifices performance is when getting a few extra percentage points means jeopardizing data integrity. There would have been no sense in developing ZFS as a replacement for existing filesystems if it did not keep data safe.

I would say that most places that are using XFS at scale are doing so with an external RAID array or SAN that handles all of the individual disks and presents a single logical LUN to the filesystem. ZFS is fairly unique in that it combines lower level disk and RAID management with a filesystem, typically these functions were handled in separate layers (like md), or in hardware.

Usually I hear people complaining about slow file creates/deletes/renames with ZFS or slowness navigating or dealing with large directories. A quick Google confirms that there are lots of people that have seen these behaviours. But I don't use it myself so I don't have any specific data. I was just trying to say that ZFS has a reputation as being a better filesystem, but not a faster one, at least with the people I talk to about storage.

ARC is a technology from IBM. Intent logs go way back. Sure ZFS is doing some interesting things, but it's a bit ironic to talk about innovation given the fact that it was more or less based on Netapp's WAFL filesystem, and all of the lawsuits that followed on from that.

I believe that the people with whom you spoke are referring to getdents() being slow when the cache is cold. That is because ZFS does not at this time implement directory prefetch logic. That will change in a future release of ZFSOnLinux, but it is a low priority because cold cache performance is not terribly important for production use.

I am beginning to think that not only do you not use ZFS, but that you have never used ZFS. I suggest that you try running your own benchmarks and workloads on it. It is a joy to use. I think you would agree if you were to use it in a blind fold comparison test.

Reply on news.ycombinator.com