Comment by 555h | Hacker News Reader

Comment by 555h | Hacker News Reader

Parent Full thread

You can probably achieve what you're looking for by stacking a few filesystems. For example, you could create a separate ZFS pool/vdev with a single full-disk zvol on each disk. Then use mdadm to create a RAID array of the zvols. Then put ext4 (or whatever) on the mdadm array.

I've done something similar for the purpose of getting FDE with ZFS in linux. It can be a little finicky, but it's definitely workable.

One ZFS-specific caveat (which may conflict with your desire to get high storage efficiency): you way need to prevent your ZFS pools from filling up too much [1]. You can either enable discard/TRIM on the whole stack, so the top level FS (e.g. ext4) can let ZFS know when a block is actually free. Or alternative just to limit your zvols to 85% (for example) of their respective pools. The latter is my preference, because there was originally a bug with discard in zfs and it's not immediately clear if it's totally fixed (although my fstrim tests seemed to work out fine).

[1] https://www.reddit.com/r/zfs/comments/3vtur4/what_exactly_ha...

Hmm, that actually sounds workable. I could even format the mdadm device as ZFS too if I really wanted. I am somewhat worried about performance, have you had any issues with that?

I haven't had any issues with performance, but then again my requirement was just "reasonable performance".

I ran some quick benchmarks (data below). Obviously this is far from rigorous, but maybe it'll be useful. In previous tests I found that volblocksize=128K was optimal for my stack -- which is why the last benchmarks use that setting.

Every additional ZFS filesystem in the stack may reduce storage efficiency (minimum free space requirements [1]; metadata & checksum overhead [2][3]) -- that's why I used ext4 as the top layer instead of another ZFS.

[1] (as mentioned before) https://www.reddit.com/r/zfs/comments/3vtur4/what_exactly_ha...

[2] https://news.ycombinator.com/item?id=14756360

[3] https://forums.freenas.org/index.php?threads/what-is-the-exa...

  Test setup:
   debian stable
   kernel 4.9.0-3-amd64
   zfs 0.6.5.9-5
   ZFS "pool": mirror with 2x 7200rpm drives

  Benchmark command:
   for i in `seq 1 10`; do sync; dd if=/dev/zero of=DEST bs=1M count=1024 conv=fdatasync; done

  zfs mirror -> dataset
   Data (MB/s): 125,115,104,135,148,170,135,151,118,119
   Mean (MB/s): 132.0
   Std.dev.: 19.9

  zfs mirror -> zvol (volblocksize=8K [default])
   Data (MB/s): 150,115,127,125,122,118,105,118,124,128
   Mean (MB/s): 123.2
   Std.dev.: 11.6

  zfs mirror -> zvol (volblocksize=128K)
   Data (MB/s): 68.5,112,115,114,94.3,85.1,83.1,98.4,120,108
   Mean (MB/s): 99.8
   Std.dev.: 16.9

  zfs mirror -> zvol (volblocksize=128K) -> luks -> ext4  (my stack)
   Data (MB/s): 130,94.4,109,139,138,125,94.9,124,134,133
   Mean (MB/s): 122.1
   Std.dev.: 16.8

edit: formatting

Can you please elaborate what are the advantages of such configuration?

The advantage is that you can get all the features you want, even though they aren't all available in one filesystem.

In my case, I wanted a reliable filesystem, RAID1/mirror support, block-level checksumming, and full-disk encryption. No filesystem provides these on linux right now. My solution was therefore to use ZFS to provide a mirrored, checksummed, reliable volume -- onto which I put a standard LUKS-encrypted ext4 filesystem. In the past I had tried the opposite (LUKS on the bare drives, then a ZFS mirror of the 2 decrypted volumes), but it was kinda annoying to manage, and I don't really need the other ZFS features (like snapshotting).

In the grandparent post, the requirement was the ability to expand the RAID volume without rebuilding (something that ZFS doesn't offer), plus checksumming and reliability (which ZFS does offer). So one option would be to use mdadm to manage the RAID array, and then put ZFS on the resultant volume in order to get checksumming.

The disadvantages are: extra complexity; extra overhead; more potential points of failure; more management hassle; etc. As soon as encryption for ZFSonlinux is stable, I'll be very happy to drop this filesystem stacking in favor of that!

Reply on news.ycombinator.com