LZFSE compression library and command line tool | Hacker News Reader

LZFSE compression library and command line tool

github.com

48 pointsingve19 comments

> It targets similar compression rates at higher compression and decompression speed compared to deflate using zlib

I don't doubt this is true, but I'd expect a little benchmark on the README as a proof. It would be interesting to see how LZFSE compares to LZ4 (https://cyan4973.github.io/lz4/).

It's tested against a wide range of other compressors in this thread: http://encode.ru/threads/2221-LZFSE-New-Apple-Data-Compressi...

The results are kind of underwhelming.

https://quixdb.github.io/squash-benchmark/unstable/ in particular paints a pretty good picture of the current landscape. It seems like ZStd is usually better than LZFSE and LZ4 is pretty competitive with LZFSE:LZVN.

I'm curious to see comparisons on a 64bit ARM cpu, since that's what Apple likely optimized it for. (I don't expect a huge difference, but I'm admittedly ignorant on this topic.)

I'd also be curious to see what the energy impact is of the various methods. LZFSE wasn't just designed for speed, it was also designed for energy efficiency.

I can't see how "power efficiency" can be different from "speed" in the context of a CPU decompressor algorithm. Care to explain?

I don't know the details, but I don't see why power efficiency has to be the same thing as speed. For example, various patterns of memory access or I/O may run at the same speed but could very plausibly have different amounts of energy consumption. Similarly, it's conceivable that different CPU instructions that run in the same time might have different energy demands. CPUs also have different speeds they run at which have different energy usage, it's possible that LZFSE might be designed to avoid putting the CPU into its highest-power mode. But as I said before, I don't know the details about what LZFSE does, I just know that there are various reasons why energy usage is not the same thing as CPU time.

Quoting the github page: "This is a reference C implementation". I read that as "not the optimized version used in Apple's libCompression."

The most interesting discussion of this compression algorithm that I've found is in a presentation from WWDC 2015 http://devstreaming.apple.com/videos/wwdc/2015/7125ovmdf36/7.... The goal is not strictly faster or higher compression ratio, but a reasonably balanced mix of the two at lower energy consumption (since the original target was iOS).

Note that energy efficiency is pretty proportional to performance, so it's very likely that a faster algorithm (e.g. LZ4) would also be more efficient.

(Back when I was working on energy efficiency, people kept trying to invent "low-power compilers", but they kept discovering that the fastest code is also the most efficient.)

Based on internet bechmarks, zstd looks like strictly better option: https://github.com/Cyan4973/zstd

Those benchmarks don't include energy efficiency, which was a primary design goal of LZFSE. I also don't see LZFSE on that page anyway, which makes it kind of hard to compare.

There was an Apple presentation that claimed lower decompression energy usage:

https://developer.apple.com/videos/play/wwdc2015/712/?time=4...

On an iOS device this is a pretty hot path, since it's used to decompress the asset catalog, including some images, so most of your apps on iOS probably already use this algorithm.

Same link posted a few days ago.

https://news.ycombinator.com/item?id=11922484

Previous (but brief) discussion.

https://news.ycombinator.com/item?id=9689471

Pardon me if this was already answered. Is this the official implementation of LZFSE from Apple? The main contributor for the Github repo @ebainville is listed in Cupertino, CA.

Reply on news.ycombinator.com