Hacker News Reader

Comment by ryao | Hacker News Reader

I am the author. Feel free to respond with questions. I will be watching for questions throughout the day.

Is there a plan at some point to include a daemon or a cron job to run automatic zpool scrubbing? I believe this was a feature that is available in other OSs' packages, but not currently with ZoL. Currently, I include two cronjobs, like so:

    18 * * * * /sbin/zpool list | grep ztank | grep ONLINE > /dev/null || /sbin/zpool status
    35 1 * * 4 /sbin/zpool scrub ztank

This way cron simply emails me if there are errors. However, I'd like a lot more communication from my storage array: if there are detected errors, I want to know right away.

My other question is much more specific to my case. Stupidly, I bought the Western Digital Green drives for my server (running in as a mirror of two drives). They are normally under no heavy load: just occasional file access to store/retrieve pictures, documents, or stream video. How likely am I to run into problems/should I replace these drives ASAP?

ZoL 0.6.3 introduced the ZFS Event Daemon. This functionality could likely be implemented into it. Please file an issue requesting this:

https://github.com/zfsonlinux/zfs/issues/new

I very much adore ZoL. Thank you for your efforts. Everything critical works and works very well.

While I get the sense that this is probably not the focus of your own work, do you have any thoughts on the maturity of the "share" facilities when using ZoL, and as the project matures will these become more of a priority? These are "nice to have" features that are obviously of relatively low importance.

You mentioned shareiscsi is unimplemented, but I also find that its friends sharenfs and especially sharesmb have some rough edges as well. When I moved a pool from an OI machine to a Linux machine, I had to massage all of my share attributes to make them function.

You can obtain the commands that you used to massage your sharenfs and sharesmb settings from `zpool history`. Please file issues with them in the issue tracker:

https://github.com/zfsonlinux/zfs/issues/new

Please include information describing your distribution, the distribution release, your kernel version, the ZoL release and also the Samba version.

I guess I am curious about how this functionality is viewed more generally / philosophically and what state is considered to be in. I wonder whether it is perceived to be good enough for production use, and whether it's expected to work as smoothly as it does under Illumos.

In my case I encountered situations where valid configurations from OI weren't supported in ZoL, but I remember coming to the conclusion that these limitations were already known and addressing them simply wasn't a priority at that time. Since this was some months ago the situation very may well have changed already!

The share support is one of the few things in ZoL that I do not actually use, so I cannot make a definitive statement on this topic. At present, the project does not have regression tests in the LLNL buildbot. The majority of the enabling work in this area was done by Turbo Fredriksson.

https://github.com/zfsonlinux/zfs/commits?author=FransUrbo

I would emailing the zfs-discuss mailing list with Turbo on CC to ask:

http://zfsonlinux.org/lists.html https://github.com/zfsonlinux/zfs/commits?author=FransUrbo

Additionally, I am aware of some SMB improvements by Turbo that are currently pending review. They are scheduled for inclusion into 0.6.4 provided that no issues are found during review:

https://github.com/zfsonlinux/zfs/pull/1476

I don't really have a question, but as a ZoL user I'd just like to say thanks for all the hard work.

It makes management of my disk arrays pretty painless and has some fantastic migration/recovery stuff going on. All of which I'm sure you know!

Seconded. ZFS is the only filesystem I trust with my children's baby pictures, as well as to store the git repo's for my personal projects (stuff I don't want on GitHub for a variety of reasons).

I am happy to hear that. While I certainly think ZFS is the best filesystem available for storing this kind of data, i would like to add a word of caution that ZFS is not a replacement for backups. I elaborated on this in one of the supplementary blog posts:

https://clusterhq.com/blog/file-systems-data-loss-zfs/#disk-...

Absolutely. I have a regular backups strategy that also backs up to a ZFS system :).

I use ZFS on a dual-boot Mac to cross-mount the Linux partitions on OS X. ZFS is pretty much the only file system that allows this: The XFS OSXFUSE plugin is read-only, the ext plugin only supports ext2 with unstable write support and BtrFS can't be mounted on OS X at all.

The ZoL-derived OpenZFSonOSX port inherits ZoL's maturity, runs in kernel space and the OS X integration is really nice (Notification Center integration in ZED, custom icons, etc).

Lovin' it!

I followed you and Brian's contributions quite closely at my previous job. We had some pretty extreme backup targets for our VPS's and the old tools were starting to become bothersome; particularly for keeping a couple hundred million files sync'd between data centers. I knew about ZFS's send/receive but we were a linux shop.. About the time we(I) were going to make some major changes to the backup systems I gave ZFS another Google, as I like to re-check my assumptions every now and then, and discovered ZoL went "stable" just that month! I immediately pushed to give it a spin and the rest was history. Learning ZFS was fantastic fun. It challenged everything I thought a filesystem was capable off. L2Arc, snapshots(cloning and shared data wut?!), ZVol's, checksum's, and on and on and on. Thanks for all your hard work and making this possible!

Can you talk about performance of virtual machine disks on ZFS? How is ZFS better/worse than BTRFS?

Before I answer that, let me say that ZFS has two options for the storage of virtual disk for virtual machines. The first is the traditional file. The second is the zvol. The zvol is a virtual block device that is a lower overhead option than a traditional file. The default internal records (blocks) vary for each. On datasets where files are stored, this is called the recordsize and it is 128KB by default. This is a per file property that is set with the value of the dataset's recordsize at the time of creation. On zvols, the volblocksize is 8KB by default. This is set at the time of creation of the zvol. In both cases, partial block writes cause a read-modify-write penalty and typically, zvols are more performant by default. At the same time, LZ4 compression seems to have the counterintuitive consequence of making the larger recordsize about equal in performance in filebench tests that I have done, so it is hard to say which is ultimately better. It is also worth noting that there are several improvements in the pipeline for ZoL zvols that should improve its performance:

https://github.com/zfsonlinux/zfs/pull/2484

With that introduction out of the way, the actual answer to your first question is that it is very dependent on your workload, so I cannot provide a solid answer, but I can discuss my own personal experience. I am involved with ZoL because I was interested in the performance of virtual storage for a home server in 2011. The technology at the time could not compare to ZoL and as far as I know, still cannot compare. In specific, I had 6x Samsung HD204UI drives connected to an AMD Phenom X6 1090T. A configuration with MD RAID 6 + LVM + ext4 did not manage more than 20MB/sec, regardless of whether I used KVM or Xen. A raidz2 configuration using ZFSOnLinux did 220MB/sec. someone with 4 disk recently had a similar experience about a week ago where MD RAID 5 + LVM + XFS and could not get more than 44MB/sec while a ZFS raidz1 configuration managed 210MB/sec if I recall correctly.

As for how ZFS is better/worse than btrfs, ZFS has several advantages in terms of its implementation. In specific, it has ARC that provides a scan resistant cache to maintain performance consistent. It has L2ARC for using flash to extend that cache. It has the ZFS Intent Log, which allows it to avoid blocking on expensive full merkle tree updates. This has allowed ZFS to outperform btrfs in ways that amazed the btrfs developers:

http://comments.gmane.org/gmane.comp.file-systems.btrfs/2754...

It has SLOG devices to allow flash to be used to accelerate ZIL. It also has a custom IO elevator that does a very good job of ensuring performance consistency:

https://twitter.com/lmarsden/status/383938538104184832/photo...

Quite honestly, here are the 5 hypothical areas in terms of where performance can be in any given comparison and what I expect the distribution to be:

1. Areas where ZFS significant outperforms btrfs. I expect there are many of these.

2. Areas where ZFS slightly outperforms btrfs. I expect that there are many of these.

3. Areas where ZFS and btrfs are equal. I expect that there are some of these.

4. Areas where btrfs slightly outperforms ZFS. I think some of these probably exist.

5. Areas where btrfs significant outperforms ZFS. I do not expect any of these to exist. If they do, they indicate bugs in the ZFS kernel driver that need to be corrected.

The areas where I think btrfs might significantly outperform ZFS today are:

1. Uncached directory lookups (getdents performance). ZFS does not currently have directory prefetching and btrfs might. This only affects cold cache performance, so it does not affect production usage and has been a low priority. It will likely be fixed in the next 12 months.

2. Small file performance. btrfs does block suballocation while ZFS does not. This should change in 0.6.4 when ZFS will begin storing small files in the dnode (the ZFS equivalent of the inode).

That said, there is nothing I or anyone can say that is a valid substitute for your own testing and I encourage you to run your own tests.

I hope that this answers your question.

What's performance like on ZoL compared to Solaris/OpenIndiana/FreeBSD?

Performance is a difficult question to answer because it is very workload dependent. However, I can say a few things on performance:

1. Solaris's EULA prohibits the publication of benchmarks without the explicit permission of Oracle. I have not done any benchmarks of Solaris and if I had done any benchmarks, I could not publish them without placing myself and/or my employer at risk of a lawsuit from Oracle.

2. Block device drivers are known to influence performance. In situations where Linux has superior driver support, ZoL will outperform its counterparts on other platforms.

3. There is a key performance fix in a pull request that is planned to be included in 0.6.4. I have seen benchmarks where it enables ZoL to outperform its counterparts on other platforms, although the other platforms were subject to inferior block device drivers. Before the fix, ZoL performed worse:

https://github.com/zfsonlinux/spl/pull/369

4. I have posted some information on performance to the ClusterHQ blog:

https://clusterhq.com/blog/state-zfs-on-linux/#comment-15844... https://clusterhq.com/blog/state-zfs-on-linux/#comment-15880...

5. A list of performance improvements in Open ZFS is available from the Open ZFS wiki:

http://open-zfs.org/wiki/Features#Performance

6. It is quite likely that I will post a future blog post discussing performance. However, performance is a difficult topic to discuss because it is extremely workload dependent. No matter what I or anyone says, or any benchmarks you see, the best measure of a storage stack's performance will ultimately be a test of your own workloads.

Reply on news.ycombinator.com