A poor entropy coding result in Zstandard

> So the theoretical minimum is ~1.2 MB. Zstd is only acheiving ~1.4 MB, which is nowhere near the theoretical limit.

> Interestingly, 7-zip compresses the data much better than zstd, reaching a size of 1,298,590 B.

This isn't surprising to me. Zstd is designed for real time compression at I/O speeds (~1 GB/s), so it probably would leave some entropy in exchange for speed, but LZMA (7zip default) is designed for good compression ratios.

Might do better if swiveled it so all the upper 8 bits are stored contiguous, then all the lower 8 bits-- zstd likes bytes.

His method for computing entropy is not so good. His method works for an independent and unpredictable sequence of bytes, but the actual data is a bunch of signed 16-bit residuals that presumably are normally distributed. Really, he should be measuring the entropy of the original heightmap, but instead he is measuring the entropy of the heightmap after he has applied a couple of confounding transformations (predictive filter, words->byte pairs, and a random shuffle) that act to increase the apparent entropy.

Calculating the exact entropy of data is at least as difficult as breaking encryption. Encrypted data should be indistinguishable from random (full-entropy) noise, but really its entropy should be the entropy of the plaintext data plus the entropy of the encryption key.

I would not write this as failure of zstd.

There are no comments on particular command line usage of respective compression algorithm frontends.

zstd has higher compression levels that near LZMA and you can additionally train a specific dictionary to be used in compressing particular kind files.

Try TurboRC using 16-bits format: https://github.com/powturbo/Turbo-Range-Coder

Wonder if he tried zstd --long=31

Or, given that he's basically mentions FCM/PPM, zpaq.

What benefit do I have with using zstd over the rest? Is there any?