Hacker News Reader

Comment by shmerl | Hacker News Reader

BTRFS is good. Surprisingly it's the only common upstream Linux filesystem with transparent compression.

Do you think the transparent compression is worth it? I imagine most of the data people have in their computers is already compressed (images, audios, videos, etc.), so it would only slow down reading and writing for most files.

I can only see it making sense if much of your disk space is used up by uncompressed text files (or other types of uncompressed filetypes).

It saves I/O on HDD, which is useful if you have a good multi core CPU. Some things are compressed but not all. BTRFS has some heuristic which switches off compression if it's not saving space. It's not very elaborate now, but they plan to improve it.

See https://btrfs.wiki.kernel.org/index.php/Compression#What_hap...

I'm using zstd compression for my btrfs volumes.

Using compress=zstd on system root saves me about 35-40%. And I save about 1/2 the space if I use compress-force=zstd mount option. I do this on computers with SDHC/SDXC cards which have dog slow writes and lots of spare CPU time, and some of the time on HDD. I don't use compression on computers with NVMe, it doesn't appear to help much at all, and also I'm pretty sure the compression is currently a single threaded pipe, so for multi-queue devices it's plausible some workloads could become slower.

Another use case, soonish I hope to see support for selectable compression levels for zstd (currently only with zlib). And also zstd supports use of a training file for even higher ratios. Both would work well for archiving and seed/sprout use cases, where taking the heavy front end hit with slow writes will be worth it (faster download times, and of course the sprout can have a different compression level or no compression so subsequent writes can be fast.

Latest squashfs support includes zstd and also selectable levels, if you need a read-only image.

Do they plan to implement multithreaded compression / decompression?

Btrfs already supports multithreaded compression and decompression. Each 128 KB block is (de)compressed with a single thread, but multiple blocks can be (de)compressed in parallel.

The zstd CLI supports multithreaded compression with the flag `-T <num-threads>`.

Good to know, thanks! Can you add that info to btrfs wiki compression page, please[1]?

1. https://btrfs.wiki.kernel.org/index.php/Compression

I implemented zstd compression in btrfs/squashfs, and work on upstream zstd.

* Btrfs compression is multithreaded, and can use up to the number of cores available on the system. * Compression might not help speed on SSDs, but it should help reduce the burn rate, if you care about that. * My intern wrote a patch to add compression level support to zstd in btrfs [1]. It should be merged upstream soon. * Slightly off topic, but grub will soon understand btrfs compressed with zstd.

[1] https://lore.kernel.org/linux-btrfs/20181031181108.289340-1-...

While that is literally true you can use Virtual Data Optimizer (VDO) in Red Hat and CentOS. It does transparent deduplication and compression. Red Hat bought it from Permabit and now it's also available in CentOS.

Reply on news.ycombinator.com