Transparent compression at IO speed for object storage
blog.min.io
blog.min.io
Zstd typically outperforms snappy, and has the benefit of a wide configurable window of compression ratio Vs speed - and that's important if you want to use a dynamic algorithm to select the compression ratio to maximize throughput.
Usually, when I see people claim things that are mutually inconsistent about compression, it turns out they have very different tradeoffs they want to do for decompression speed {avg,worst}/compression speed {avg,worst}/compression ratio.
https://community.centminmod.com/data/attachment-files/2019/...
Yes, though decompression speed (+memory to some extent) should also be considered. Also different input types has different characteristics, so a single type isn't very representative.
While ultimate compression efficiency is interesting, in practice I find that most often you are looking for a "good enough", meaning "give me the best at xxxMB/s". For most purposes (de)compression at 50MB/s is just too slow to bother.
For a server anything considerably below 1GB/s/core is too slow, otherwise you will too easily saturate your CPU before your NIC or disks are saturated.
Also, zstd seems to operate over the widest range - a simple algorithm to use that would be: "Each second, check... Is network utilization over 90% and CPU utilization under 90%? If so, increase compression level by one. If the reverse, decrease compression level by one"
Zstandard is good, no doubt. I didn't include compressor with entropy compression, except gzip which I added as a reference point for the ratios. After all I know it intimately after porting it to Go.
In overall terms, it does have an unavoidable bigger cost both for compression and decompression. So to have the smallest possible impact we have chosen not to go that direction (yet).
We might add it as a lifecycle option, where it compresses to zstd when the object reaches a certain age.
Furthermore the initial implementation was Snappy, so switching to S2 could be done seamless without requiring different decompressors and keep the implementation simple.
After all, the most effective way would be for the uploaded data to be compressed. The on-server compression is to catch the cases and compress them "reasonably" without taking too many resources.