Facebook/Zstandard – Fast real-time compression algorithm | Hacker News Reader

Facebook/Zstandard – Fast real-time compression algorithm

github.com

56 pointsanimationwill13 comments

ZSTD seems to be relatively revolutionary in the field of compression, at least in most of the test cases I've tried. Binaries typically compress as well as LZMA2, averaging about a 50% compression ratio, with nearly an order of magnitude faster decompression.

ZSTD beats DEFLATE and Gzip seemingly in every metric. It decompresses faster than almost any algorithm out there, save maybe LZO or LZ4, and has a compression ratio comparable to much slower algorithms.

Arch Linux switched to ZSTD for compressing their package database at the beginning of this year, and only increased the overall average compressed size by 0.8%, while speeding up decompression several orders of magnitude compared to LZMA2.

It was awesome for a use case I had. When sending some raw data, the application was network-bound. When we GZIPd the data, we were horrifically CPU-bound. When we tried ZSTD, it actually made our data rates manageable

I’m hoping browsers adopt this as a compression standard too, just like Gzip and Brotli.

As it happens, [I was adding Zstandard support](https://github.com/kafkajs/zstd) to [KafkaJS](https://kafka.js.org) yesterday. Sadly, the closest thing there is to official bindings for NodeJS haven't been updated for 4 years and don't build against newer versions of Node, so I ended up having to rely on some random fork that I have no reason to believe is going to be maintained.

Unfortunately, the overlap of people that know C++ well enough to write high quality bindings and also have an interest in NodeJS is not that large. I tried to get into it myself, but there's a pretty steep learning curve and once you go beyond tiny toy examples it gets pretty hairy.

If someone has the skills and interest in taking on a medium sized open source project that would be immediately useful to a lot of people in the Node community, Zstandard would be a good choice.

Hey I made the original feature request to add support for Zstd on the KafkaJS repo about 2 years ago! I no longer work at a place that uses Kafka, but I just wanted to say KafkaJS was a brilliant piece of software and IMHO the best Node.js library for working with Kafka!

Thank you, I appreciate that!

> Zstandard is dual-licensed under BSD and GPLv2.

Both of which don't offer a patent license.

How well does it perform in WebAssembly?

There's a web assembly version in the repository : https://github.com/facebook/zstd/tree/dev/contrib/single_fil...

Could be worth checking

I just would like for wireshark or tshark to handle zstd- or lz4-compressed pcaps. Then all would be good.

so hooli wasn't google after all

If I remember correctly when Judge was asked about what company hooli was supposed to represent he said something along the lines of “hooli is not one company, it’s all of them [FAANG]”.

Yeah but what's its Weissman score?

Reply on news.ycombinator.com