Maybe fio is producing trivially compressible data? Dedup? Directio not working?
What am I missing?
The initial file was created with fio rw=randwrite, so the dummy data is random.
I think its even more impressive that ZFS can do this. I verified with ext4 fs on a single NVMe drive and no matter what knobs I turn, I am getting around around 3.5 GB/s.
Edit: See note 2 at the end of the article, I posted the raw results of another 100GB file test.
Edit 2: See note 3. I rebooted the VM with 2GB ram (instead of 64GB) and it matches almost exactly as per your prediction ~ 20GB/s. But, it still doesn't explain the absurd 80GB/s sustained reads of a 100GB file with only 64GB ram.
I'm thinking that a) fio might create test data that is "trivially" compressible and b) this might occur before directio says "ignore the buffer cache and write directly to the drive", but then what gets written is 10x or 100x smaller.
Another possibility is that somehow directio doesn't work when zfs+compression is turned on?
I'd do a test where everything else is the same, but turn lz4 compression off. In theory, with a real world dataset, compression will cause CPU load to go up, but I would not expect to see the datasize at the disk go down by more than 2x to 3x, and therefore the bandwidth should not go up my anymore than that, and only if the CPU isn't the limiting factor. In most cases, compressed (or encrypted) filesystems should be slower, not faster because of this.
So a benchmark without compression would be the first step. Then maybe as a second step, a "real world" benchmark - copy 10-100GB of representative data or something like that - with and without compression to see if bandwidth goes up or down and how bad CPU utilization goes up.
Another estimate is to look at the stats that tell you what the compression ratio is. If it's really, really high, that just means fio created all-zeros, really fast.
???
See note 4: https://neil.computer/notes/zfs-raidz2/#note-4
At the end of the day, I found that with ZFS it was just very difficult to bypass, avoid, turn-off, etc. all the caching. The machine I was using has a lot of RAM. I had to reduce the RAM to like 64GB with a boot option to even see what was happening. I was still never thrilled with the numbers I got.
Then, finally it dawned on me: with enough RAM, and an efficient filesystem like ZFS, the underlying storage is only a small part of the equation. The important thing is that your benchmark is accurate to the real world workload. If the real world can use a lot of buffer cache and RAM, and it makes it go fast - then good, use it like that. Which I did, and was happy, and forgot all about it.
What we want to avoid is fio saying you have 100GB/s and then in the real world you don't see nearly that and are disappointed. To me benchmarks serve two purposes: comparison and prediction of performance with a reproducible test case, for example compare the effect of ARC cache size setting. Second, diagnosing whether the hardware is working correctly and as close to ideal.
So I did see some stuff vaguely like you are seeing. I remember being impressed at ZFS and how hard it is to benchmark because it tries so hard to cache efficiently (or whatever). Unfortunately it was like a year ago so I don't remember more details.
I also did the following to drop (most) cached data before the test:
sync; echo 3 > /proc/sys/vm/drop_caches; zpool export $pool; zpool import $poolUpdate:
Ok, so I used the ':' syntax in the --filename option, I got something that makes sense. FIO created invidual files for each thread. I test numjobs: 4, 8, 16 and it plateaus around 8 jobs. Adding additional files+jobs actually makes things worse.
neil@ubuntu:~/mnt/disk$ ls
dummy0 dummy10 dummy12 dummy14 dummy2 dummy4 dummy6 dummy8
dummy1 dummy11 dummy13 dummy15 dummy3 dummy5 dummy7 dummy9
neil@ubuntu:~/mnt/disk$ sudo fio --name=read_test \
> --filename=/home/neil/mnt/disk/dummy0:/home/neil/mnt/disk/dummy1:/home/neil/mnt/disk/dummy2:/home/neil/mnt/disk/dummy3:/home/neil/mnt/disk/dummy4:/home/neil/mnt/disk/dummy5:/home/neil/mnt/disk/dummy6:/home/neil/mnt/disk/dummy7:/home/neil/mnt/disk/dummy8:/home/neil/mnt/disk/dummy9:/home/neil/mnt/disk/dummy10:/home/neil/mnt/disk/dummy11:/home/neil/mnt/disk/dummy12:/home/neil/mnt/disk/dummy13:/home/neil/mnt/disk/dummy14:/home/neil/mnt/disk/dummy15 \
> --filesize=20G \
> --ioengine=libaio \
> --direct=1 \
> --sync=1 \
> --bs=512k \
> --iodepth=1 \
> --rw=read \
> --numjobs=16 \
> --group_reporting
Updated the article: https://neil.computer/notes/zfs-raidz2/#note-5
actually, I think those numbers are pretty good. you can zpool export to unmount and then run a non-destructive read test with --filename=/dev/nvme0n1:/dev/nvme0n2... and check that the raw devices get, say, 80%-99% of the advertised speed.
Then you can create a "raw" zvol (no filesystem) with:
create -b 128K -V 100G tank/testme
Running through the ZFS filesystem seemed to slow down by a good 50% of the ideal. None of this has been tuned.
So to see 20GB/s+ seems pretty good to me.
Since my machine is a build server, my benchmark for that is:
git clean -dxf; make defconfig
time make -j vmlinux
https://news.ycombinator.com/item?id=25956670
Had to use direct I/O and use random large IO patterns to avoid various caching, buffering and prefetching effects.
10 PCIe 4.0 SSDs give you maximum of around 70GB/s.
The test show sustained throughput of 80GB/s without compression. Which is still higher than the maximum allowed by 8 x 4 PCIE 3.0 slot at total of 32GB/s.
Waiting for some ZFS expert to give a plausible explanation.