Show HN: New compression algorithm beats ZSTD by 14%

github.com

1 pointseccode3 comments

Hi all, I made a data compression algorithm that works as follows: Replace common bigrams with a fixed value: "of the" => "of e" "and she" => "and e" "when there" => "when e"

To decompress, look up the bigram dictionary for the right value of e.

I've had some weird bugs in the decompression process, so I'm looking forward to others looking at this.

Open articleView on HN

What kind of data is it 14% better on?

The algorithm code looks like it's actually using Zstd, is that right?

Please show some benchmarks. I'd like to repeat your test!

I tested on the dickens dataset (Charles Dickens novels). This algorithm works well on English text. I do not expect it to be any better than zstd at binary data. I haven't tried code.

It does used zstd. My algorithm is a preprocessing step for zstd.

Commit that data + test code :)

Add the results to the readme ::))

Reply on news.ycombinator.com