Hacker News Reader

Comment by zaarn | Hacker News Reader

zstd-fast is almost as fast as LZ4 and gets a tiny bit better compression, though not a lot of software supports that mode. ZFS does, hence I have some numbers to back that up.

View on HN

I just had a little look into these somewhat new zstd options --fast 1 to fast 5. fast 5 does seem almost equivalent to lz4 performance in compression ratio and compression 'speed' (or cpu load) except decompression 'speed' is still almost half of lz4, so at that extreme of zstds options it is squarely beaten by lz4. But in terms of compression ratio - which in most cases should be the more crucial characteristic for any compressor than cpu load - this setting seems to be well under zstd's knee of diminishing returns. --fast 1 has about 15% more cpu load for about 112% of compression performance than --fast5. And then zstds lightest non-fast setting (zstd-1) for 50% more load in compression and decompression, compresses 32% better than --fast5 (and lzo and lz4).

The decompression load is an eye catching feature of lz4, but its compression ratio is not just marginally compromised, it is a much less powerful compressor in that prime characteristic than zstds weakest non-fast setting. I can appreciate why Btrfs is in no hurry to include lz4 while it supports zstd:1 to 15. Although it would be interesting to test if lz4 could have a niche on fast storage combined with slow cpus.

There are use cases for lz4, I've had one in an embedded project. Slowish SoC (i.MX6 DualLite IIRC), relatively fast storage (eMMC - these tend to perform better than regular MMC). I measured boot time for the available kernel compression options and lz4 won by 100-200 ms or so compared to the next best option, which may have been gzip. It didn't make a big dent in overall boot time, but not bad for just setting an option. zstd, if available, which it wasn't, would have gotten a marginally worse result probably. IIRC, lzma was slower than uncompressed.

gzip would seem to be a surprising runner up as its decompression cpu load is reportedly around 3-4 times as heavy as zstd:1 with a similar compression ratio. I would not rule zstd out, it seems to better every alternative given the right level setting, except lz4 which is the outlier but fixed at a relatively low compression level. As the high performing outlier it tantalisingly suggests room for improvements :]

At least older Linux (the kernel) versions don't support zstd compression (for the kernel image itself), that's why it was not part of the experiment.

I read it was only mainlined last year following this thread[1] where a kernel dev presented benchmarks and advised "zstd is an improvement on almost all metrics"

Of course zstd somewhat of a portfolio project of facebook developers, and they seem to have integrated it quite nicely into Btrfs to atone for their debt to society - I'll take that :D

[1] https://lore.kernel.org/lkml/1588791882.08g1378g67.none@loca...

Reply on news.ycombinator.com