I haven't investigated either approach, but this is one of my next projects I'm working on, so I'll be figuring it out either way. I was wondering, does Zstd provide any ability to serialize/restore and continue where it left off?
(Context: Disk compression. Due to long-term factors beyond my control I've had severe disk space issues for over a decade (with frequently only tens of MBs free). Complex story ending in "cannot work". I recently realized that some of my disks contained data I didn't immediately need, and began wondering about ways I might be able to compress this data out "of the way" so I could carefully partition the remaining space. This would need to be done in-place as I would have nowhere with enough space to hold the compressed data, at least not to begin with.)
The last time someone gave me an old empty 320GB HDD they weren't using, one of my own disks started clicking about a week later and I was able to save everything on it. I still shake my head at that perfect timing.
Of course, this meant I lost all the additional free space, haha. One step forward, two steps back...
Heavy mmap usage (VMWare sometimes uses an mmap to hold a guest's memory, for example) doesn't show up as memory usage in any system monitoring tools, and the system starts to thrash long before apparent memory usage gets high. Maybe someone here has a solution to that?
The long range matcher has a configurable match size, where it will only look for matches that are at least that large. By default it is 64 B, but by making it larger, say 4 KB, you can ensure that if you are going to force a page fault, you get enough benefit.