Hacker News Reader

Nice post — one suggestion would be to use zstd instead of gzip though. Out of the box it is much faster and will result in smaller files, and allows tuning to fit inside memory and cpu thresholds different use cases might prefer.

Friends don’t let friends use gzip :)

View on HN

Thanks, I thought about using a different compression tool but everybody knows gzip so I figured I’d keep it simple. Litestream uses LZ4 internally (which is great) but it’s not something a lot of developers come across too often.

Storage is so cheap nowadays that usability and compression speed are more important for most people.

For a given compression ratio, zstd compression is faster than gzip by a good margin (even without using multiple threads). And zstd decompression is pretty much always faster than gzip too.

Here is a nice page for mapping "what first came to mind" to "what's actually fastest in 2021": https://jolynch.github.io/posts/use_fast_data_algorithms/

TL;DR, lz4 is 10x faster than gzip (though does not produce as small as a file), zstd is twice as fast and better.

Reply on news.ycombinator.com