Comment by crazygringo | Hacker News Reader

Comment by crazygringo | Hacker News Reader

I'm actually surprised at how insanely terrible the "compression" is.

Looking at the table at the end, I'm not surprised at all that the "weird" obfuscated code is ~2000x the size of the original source.

But I am surprised that that the gzipped weird code is still ~25x the size of the original source, as opposed to ~0.25x for gzipping the original source.

After all, the amount information in the weird code should still ultimately be approximately the same as the original source code, right? Or maybe double or something like that. I'm very surprised it's twenty-five times as much.

The only reason I can guess is that the "weird" process results in information structures that are represented in an extremely hierarchical way, and gzip is built for stream compression, and is unable to find/represent/compress hierarchical structures?

And if that's the case, it makes me wonder if there are any compression algorithms which are able to handle that better? That might not be based on "dictionary words/sequences" as much, but rather attempting to find "nestable/repeatable syntax patterns"?

Compression is all about detecting patterns in the data, usually with some kind of sliding window of past data, used to predict future data. For gzip in particular, this sliding window has a fixed size of 32KB.

If the "weird" version is ~2000× the size of the original source, gzip would need a sliding window of ~2000× the size (about 64MB) to obtain equivalent compression.

Also, an encoder like these can inject any amount of entropy (eg replace "0" with "100500999-100500999") which gzip can reduce to "100500999-same" but without knowledge of JS semantics has no chance to reduce to either "0" nor the nearby "172913211-172913211")

> For gzip in particular, this sliding window has a fixed size of 32KB.

I'm surprised people keep reaching for gzip for this sort of comparison. LZ4 / zstd compress better at the same compression speed, and zstd and brotli compresses much better. Brotli is also supported in all modern browsers.

There's basically no reason to use gzip in new software. (With the exception of compressing for the web - at least until zstd makes it into browsers.)

In this case, op explicitly compiled javascript for use in web browsers, so in my opinion reaching for gzip is not so strange. There is no browser support yet except behind feature flags in Chrome, and in curl, but that's not a browser. Also, zstd was explicitly designed to compress as good as deflate but much faster. You can also use zstd to compress 'better' but that costs some CPU time again but that is mostly a different use case

Brotli is a much better choice for compressing JavaScript for the web. It’s supported by every browser (and curl I think), and the compression ratio is way better. The only downside is that you probably want to compress once instead of on every request, because brotli compression is slow. Easy enough to set up though.

It would be interesting to see what better compression schemes could do here.

> After all, the amount information in the weird code should still ultimately be approximately the same as the original source code, right?

Taking a very high level view of this: the compressor can't know about the rules that are the difference between a .weird file that is valid .js and a byte sequence that is not and neither can it know the difference between bytes that are valid .js and look like .weird and are part of the subset that are encodings of a .js file and those that are not.

The compressor builds a more or less accurate model of observed probabilities, but even the most optimal encoding would still have to keep escape hatches around for all other bit sequences. Those escape hatches are not free.

~100x compression (from weird to weird.gz) is really good, no matter the input. Simple compressors like gzip are limited in what they can do. They have size-constrained dictionaries, cannot have symbols of less than one bit, etc... Gzip works fine with real life data, and does it in a reasonable amount of time using a reasonable amount of memory, but it may not fare well with such a torture test.

New, fancy, and really slow compressors with their contexts, arithmetic coders and neural networks will give a much better result, maybe close to ideal, but it will take orders of magnitude more time and use up GBs of memory doing so.

For comparison, 7-zipped modernizr.weird.js is only 17.85KB which is almost exactly 2½x the size of the original.

Ah thank you! That feels far more sane.

I don't think I've ever seen such a dramatic difference in compression rates before. It's fascinating to see.

Reply on news.ycombinator.com