Hellishly Slow Level 13 Deflate Compression
kirill.korins.ky
kirill.korins.ky
It would be interesting to see some economics of what 8,000% increase in encoding time takes to make that money back in terms of storage or bandwidth. I also wonder how brotli/lzma would compare here. Are there some obscene modes on those that had similar results?
I once used https://github.com/google/riegeli and a low zstd compression level to store large quantities of protobuf data in an efficient manner (in terms of CPU, RAM and streaming to disk). Shame Riegeli is not well known, not well documented and does not have many tests.
Far better, just like anything else based on arithmetic coding. The main distinction here is that the output can still be decompressed with a standard Inflate implementation.
This class of compression programs sees larger differences due to the way the data is modelled instead of the specific entropy coder used.
It's entirely possible the degradation of their RTG power sources would be more expensive doing the compression then just sending the data as is.
And you're eating into a limited overall power and weight budget to do rather then say, run the science on the probe.
...and of course it's written by someone with a Russian name, and has that characteristic style common to many other articles about data compression.
"OpenZL delivers high compression ratios while preserving high speed, a level of performance that is out of reach for generic compressors. OpenZL takes a description of your data and builds from it a specialized compressor optimized for your specific format." "OpenZL to offer 10% faster compression speed and 70% faster decompression speed compared to Zstandard level 1 on the Silesia corpus in our benchmarks."
"OpenZL now ships its own LZ codec, exposed as ZL_GRAPH_LZ, and the serial profile in zli. It is still being actively developed to expand its feature set and improve performance on small inputs."
(~ OpenZL-AI-LLM recognises the data structure, then guides OpenZL toward the best lossless compression path )
2. Spend orders of magnitude (literally) more on compute to run the LLM on the data than any compression algorithm would ever take.
"The unreasonable effectiveness of our first foray into training leads us to believe that the graph model is uniquely positioned to facilitate ML-guided generation of compressors. We are tempted to view this as “the next big thing” in production-scale compression. Whereas compression research has up to now eluded those without domain expertise, we believe the future of application-specific compressors will be unlocked via investment in automated learning methods."
And for decompression, the effect on memory usage and timings?