Hacker News Reader

mhx77on DwarFS – Deduplicating Warp-Speed Advanced Read-Only File System

FWIW, I've updated the comparison using the latest versions of EROFS and DwarFS. There definitely have been improvements to EROFS in the meantime. DwarFS is still orders of magnitude faster in creating the file system and can achieve significantly better compression, but throughput is very similar unless you start optimizing the EROFS image for better compression.

mhx77on DwarFS – Deduplicating Warp-Speed Advanced Read-Only File System

Thanks for the feedback! I was probably a little too excited when writing the documentation. Reading it again with some distance, I can certainly see why this might be a bit off-putting. I'll keep this in mind for the future!

mhx77on DwarFS – Deduplicating Warp-Speed Advanced Read-Only File System

https://en.wikipedia.org/wiki/Locality-sensitive_hashing https://en.wikipedia.org/wiki/Nilsimsa_Hash https://github.com/mhx/dwarfs/blob/main/doc/mkdwarfs.md#nils...

mhx77on DwarFS – Deduplicating Warp-Speed Advanced Read-Only File System

Correct, there are no images in the data except for 68 PNGs. It's just HTML files.

mhx77on DwarFS – Deduplicating Warp-Speed Advanced Read-Only File System

It's ever so slightly smaller than a .tar.xz of the same data. The main difference being that you don't have to fully extract it in order to access the data.

mhx77on DwarFS – Deduplicating Warp-Speed Advanced Read-Only File System

That's definitely a fair point! DwarFS actually works perfectly fine on 64-bit ARM and would likely work fine on a smartphone as well. Still (and I might be completely wrong about this), I think the primary goal of EROFS is to consume as little resources as possible when the file system is accessed and to be able to run on much less capable hardware, including 32-bit systems. DwarFS primarily cares about maximizing compression as long as it doesn't negatively impact performance (access times and throughput). This involves a certain amount of caching / pre-fetching and assumes there's plenty of memory available. This can be configured to a certain extent, but my pessimistic assumption is that DwarFS would use significantly more memory than EROFS. Might be worth actually backing this by numbers! :)

mhx77on DwarFS – Deduplicating Warp-Speed Advanced Read-Only File System

I've got a (relatively old) snapshot of the English Wikipedia that I'm using for testing. The snapshot is around 200 GiB in 14,000,000 files and compresses down to an 11 GiB DwarFS image.

mhx77on DwarFS – Deduplicating Warp-Speed Advanced Read-Only File System

https://github.com/mhx/dwarfs?tab=readme-ov-file#with-erofs

It's been a while since I did that comparison, so the results could be significantly different now.

mhx77on Elfshaker: Version control system fine-tuned for binaries

Somewhat related (and definitely born out of a very similar use case): https://github.com/mhx/dwarfs

I initially built this for having access to 1000+ Perl installations (spanning decades of Perl releases). The compression in this case is not quite as impressive (50 GiB to around 300 MiB), but access times are typically in the millisecond region.

mhx77on DwarFS: A fast high compression read-only file system

> mkdwarfs crashed with recursive links (1-level, just pointing to itself)

That's odd, it shouldn't crash with links at all, as it doesn't actively follow links. Can you please file a bug if you can reproduce this?

> and when I removed dirs while running mkdwarfs, which were part of of the input path

I guess this is fair, but I'll try to take a look anyway. :-)

> On success, mkdwarfs needed 1 hr, and reduced 219 dirs to a size of 970 MB. Not just source files, but also the build and install object files.

My 500 MB image with the 1100+ perls is just installations, from which I've actually removed libperl.a as I've never needed it and it really bloats the image. I've got a separate image with debug information (everything built with -g in case I need to debug the binaries), so the binaries in the main image are essentially all stripped. If I need to debug, I'll just mount the debug image as well, which contains the source files and the stripped debug data.

> 1 hr is a lot, but just think how long squashfs would have needed.

It might be worth trying a lower compression level, especially if you find that mkdwarfs is CPU bound and not I/O bound.

mhx77on DwarFS: A fast high compression read-only file system

Yes, and I've documented this now:

https://github.com/mhx/dwarfs/blob/main/doc/dwarfs.md#settin...

mhx77on DwarFS: A fast high compression read-only file system

> have you investigated why this might be the case?

Very briefly. It looks like clang has a different strategy breaking up the code (which is mostly C++ templates) into actual functions vs. inlining it, and the hot code ultimately performs fewer function calls with clang than it does with gcc. But this is nowhere near a proper analysis of what's going on. :)

mhx77on DwarFS: A fast high compression read-only file system

> If you upload a module to CPAN, you automatically get it tested against a huge matrix of configurations.

Very true, and it's definitely a great service!

However, the set of versions/configurations is still limited, and it can take an awful lot of time for the matrix to fill up. I've fixed a bug specific to perl-5.10.0 about a week ago and so far the module hasn't been picked up by that version again.

So while this is definitely good as a service for the general public, it doesn't get you very far if you're trying to build a thing that's supposed to ensure compatibility for other Perl modules across 20 years of Perl history. :)

mhx77on DwarFS: A fast high compression read-only file system

This is really cool, I'll give squashfs-tools-ng a try!

> Does the Raspbian image comparison compare XZ compression against SquashFS with Zstd?

That's correct. It's not an exhaustive matrix of comparisons.

> Also, is there any documentation on how the on-disk format for DwarFS and it's packing works which might explain the incredible size difference?

The format as of 0.2.0 is actually quite simple. It's a list of compressed data blocks, followed by a metadata block (and a schema describing the metadata block). The metadata format is implemented by and documented in in [1].

There are probably 3 things that contribute to compression level:

1) Block size. DwarFS can use arbitrary block sizes (artificially limited to powers of two), and uses a much larger block size (16M) by default. SquasFS doesn't seem to be able to go higher than 1M.

2) Ordering files by similarity.

3) Segment deduplication. If segments of files overlap with previously seen data, these segments are referenced instead of written again. The minimum size of these segments can be configured and defaults to 2k. For my primary use case, of the 47.6 GB of input data, 28.2 GB are saved by file-level deduplication, and another 12.4 GB by this segment-level deduplication. So before the "real" compression algorithms actually kick in, there are only 7 GB of data left. As these are ordered by similarity, and stored in rather big blocks, some of the 16M blocks can actually be compressed down to less then 100k.

[1] https://github.com/mhx/dwarfs/blob/main/thrift/metadata.thri...

mhx77on DwarFS: A fast high compression read-only file system

Precisely.

mhx77on DwarFS: A fast high compression read-only file system

Files that you've accessed will be kept in the kernel's cache. The cache I was talking about is a cache for decompressed blocks. Single files can stretch across multiple blocks, so you need to be able to keep more than one in memory anyway. However, decompressed files are kept in the cache in the hope that further (or even concurrent) reads will access the same blocks. Taking the example from the README where over a 1000 perl binaries are being executed concurrently, that cache typically has hit rates of 99+%:

  $ dwarfs perl-install.dwarfs mnt -f
  23:02:42.673390 dwarfs (0.2.1)
  23:02:42.676663 file system initialized [1.94ms]
  23:02:49.210158 blocks created: 226
  23:02:49.210189 blocks evicted: 194
  23:02:49.210216 request sets merged: 123
  23:02:49.210241 total requests: 50056
  23:02:49.210270 active hits (fast): 1515
  23:02:49.210293 active hits (slow): 833
  23:02:49.210318 cache hits (fast): 47482
  23:02:49.210343 cache hits (slow): 0
  23:02:49.210392 fast hit rate: 97.8844%
  23:02:49.210417 slow hit rate: 1.66414%
  23:02:49.210441 miss rate: 0.451494%

For example, reducing the cache size from 512M (default) to 32M increases the time it takes to run 1139 binaries from 2.5 seconds to almost 40 seconds.

mhx77on DwarFS: A fast high compression read-only file system

> @OP: Can you please explain why you keep 50 gigs of perl around? :-)

Sure. I've been the maintainer of a perl portability module (Devel::PPPort) for a long time and every release was tested against basically every possible version (and several build flag permutations) of perl that was potentially out in the wild.

mhx77on DwarFS: A fast high compression read-only file system

> How opposed would you be to this being reworked to being able to be mainline kernel support too?

I don't see any way of getting this anywhere near the kernel without a full rewrite. It's C++ and it depends on libraries that aren't even shipped by a lot of distributions (folly & fbthrift). And, tbh, I don't see much benefit given that FUSE these days doesn't seem to be significantly worse in terms of performance.

> I'd need to setup a full build server/system though because I need to regularly update them with new modules.

Overlay the mounted read-only fs with a read-write fs. Then you can install modules as you like and if you want to start fresh, just throw away the read-write fs. That's what I've done in the past.

mhx77on DwarFS: A fast high compression read-only file system

Author here :) I'm not sure low-spec hardware is necessarily the best use case for DwarFS. It doesn't necessarily use less resources than SquashFS, although it can create file systems that are smaller with much less CPU resources. However, it'll still need a reasonable amount of memory at run time to cache active, decompressed blocks.

mhx77on DwarFS: A fast high compression read-only file system

Hahaha, I haven't actually been coding on this for that long, it's more that I coded for a few weeks back in 2013 and only found the motivation to resurrect the whole thing a few weeks back.

mhx77on DwarFS: A fast high compression read-only file system

Read-only media, for example. Or in general, stuff that doesn't really change. In my case: https://github.com/mhx/dwarfs#history

mhx77on DwarFS: A fast high compression read-only file system

Author here :)

I'll add more benchmarks, this is still WIP and so far I've mainly tried to satisfy my own needs. My intention with DwarFS wasn't to write "a better SquashFS", but to make it better in certain scenarios (huge, highly redundant data) than SquashFS. SquashFS still has the big advantage of being part of the kernel, which makes it a lot more attractive for things like root file systems.