Comment by aidanhs | Hacker News Reader

Comment by aidanhs | Hacker News Reader

Parent Full thread

First off, I love zstd and thanks for your work - I've used it multiple times with great success.

My question is...what's up with zstd compression levels? It seems to be impossible to find documentation on what the supported compression levels are, and then there are magic numbers that make things more confusing (I think last time I checked, 0 means 'default' which translates to level 3?)

My notes when I was trying to track through the minimum compression level through multiple header files seemed to indicate it's MINCLEVEL...which appears to be -131072? But it starts talking about block sizes and target length, and I'm not entirely sure why they would relate to a compression level.

  # define MINCLEVEL  ZSTD_minCLevel() [0]
  int ZSTD_minCLevel(void) { return (int)-ZSTD_TARGETLENGTH_MAX; } [1]
  #define ZSTD_TARGETLENGTH_MAX    ZSTD_BLOCKSIZE_MAX [2]
  #define ZSTD_BLOCKSIZE_MAX     (1<<ZSTD_BLOCKSIZELOG_MAX) [3]
  #define ZSTD_BLOCKSIZELOG_MAX  17 [3]

[0] https://github.com/facebook/zstd/blob/550410d05d7c7815b1ff41... [1] https://github.com/facebook/zstd/blob/550410d05d7c7815b1ff41... [2] https://github.com/facebook/zstd/blob/550410d05d7c7815b1ff41... [3] https://github.com/facebook/zstd/blob/550410d05d7c7815b1ff41...

Basically, is there a decent reference I can be looking at here?

Thanks for the feedback! I've opened an issue to track this [0]

* Levels 1-19 are the "standard" compression levels.

* Levels 20-22 are the "ultra" levels which require --ultra to use on the CLI. They allocate a lot of memory and are very slow.

* Level 0 is the default compression level, which is 3.

* Levels < 0 are the "fast" compression levels. They achieve speed by turning off Huffman compression, and by "accelerating" compression by a factor. Level -1 has acceleration factor 1, -2 has acceleration factor 2, and so on. So the minimum supported negative compression level is -131072, since the maximum acceleration factor is our block size. But in practice, I wouldn't think a negative level lower than -10 or -20 would be all that useful.

[0] https://github.com/facebook/zstd/issues/3133

We're still reserving the right to fiddle around with the meaning of our negative compression levels. We think that we may be able to offer more compression at the same speeds by completely changing our search strategies for very fast compression speeds. But, there is only so much time in the day, and we haven't had time to investigate it yet. So we don't want to lock ourselves into a particular scheme right now.

> They allocate a lot of memory and are very slow.

Slow and takes memory to compress, uncompress, or both?

Both.

They are only slow to compress. They are as fast to decompress, or even faster sometimes. Source: I tested.

Yeah thats correct. I'll just point out that they use a larger window size, so they will use more memory to decompress, but will still be fast.

This is exactly what I was hoping for! If you just copied and pasted this into the documentation directly, that'd be more than enough. Thanks for writing it out so clearly and creating the issue.

Reply on news.ycombinator.com