Hacker News Reader

Comment by bearjaws | Hacker News Reader

People keep reinventing JSON trying to beat gzip, and it never beats the simplicity of gzipping JSON...

comets.json:

        JSON: 51949 bytes
        JCOF: 37480 bytes (0.721x)
  JSON ZIPED: 15178 bytes

Zipped json wins again.

Now I am curious – how does zipped JCOF turn out? I know that’s not the author’s intention, but I’m curious how it compresses

It's nearly irrelevant, given that I can gzip JSON and send it to literally anything built in the last 10+ years. It's so ubiquitous now that many web servers will just accept gzipped content without even exposing that fact to the back end servers. You can get all the benefits of highly compressed JSON and never have interacted with gzip.

If it could somehow produce a significant reduction with fewer CPU cycles, there may be a really niche use case, but I don't see those often.

Oh, it is completely irrelevant! I just want a data point for my mental model of gzip performance. Specifically, I am curious if the optimizations (object tables and separator elimination) are made irrelevant by gzip's dictionary, or if they result in better compression

In my experience it should not make much or any difference.

I had devs try to "optimise" the JSON our services were returning by doing things like abbreviating field names that were repeated very often or replacing long string constants with short integers or removing formatting, and so on. When I asked them to do actual test when the service returns gzipped output and they found out that even reducing the JSON size this way by half does not make any measurable difference on the compressed stream in most cases.

I found the same re: compressed stream, however I also found that hand optimizing the decompressed JSON made for significant performance improvements on underpowered devices (e.g. some old Android phone).

Well, that's true. But if performance is more important to you, just skip it and go for binary formats that can be orders of magnitude more efficient.

In general text formats are a compromise. They are bulky and and inefficient to parse. What you get in exchange is ease of development.

If you are willing to complicate the format just to improve performance, at some point you no longer get ease of development and you should just switch to binary altogether.

And zipped/unzipped msgpack for that matter.

Precisely, I'd rather use msgpack or CBOR if it requires additional libs. I use JSON because of browser support.

This actually does. I used Python gzip to see how the minified would compress and came up with 157 bytes to JCOF's 134. Using the same mechanism to gzip the JCOF saves only 4 bytes. I'm actually rather impressed.

I think people would reject your samples because the data size is too small. Production JSON comes in all shapes and sizes and situations where you care about performance usually indicates much larger payloads.

If your egress is paramount, you should of course benchmark everything to shreds. Then pick what fits.

It might on small files but it doesn't on (some) bigger ones. e.g. I dumped a Minecraft region file into JSON giving 85M. `jcof` turns this into 66M. `gzip` on the original json gives 14M for `-1` and 11M for `-9`. `zstd` also gives 11M.

But because the Minecraft JSON is highly redundant, I also tried it on 84M of fake social graph data. `jcof` gave 44M, `gzip -9` gave 19M and `zstd` gave 18M.

In summary, neat idea, but you're maybe better off with `gzip` or `zstd` in the real world.

JCOF and gzip are not mutex.

No, but when you get ~95% of the benefit (per other people's data) while keeping all the flexibility, compatibility with everything under the sun, and native-performance parsing everywhere, the supposed improvement looks a lot less like one.

A sibling has the data. For that case it's 10% of egress which can be a healthy chunk of change on your bill. For all such use cases you should of course benchmark a lot and then pick what fits. I think this tool is a nice one to have in the toolbox. The low-rate files seem to be close to optimal.

> A sibling has the data.

And now has updated data showing that sorting your keys lets `lzma` beat `jcof|lzma` at every level. The egress bill has been reduced! All rejoice!

EDIT: Nope, I was wrong, I was doing `lzma` against `jcof|zstd`. `jcof|lzma` is still sneaking ~1M below `lzma` at all levels.

> For that case it's 10% of egress which can be a healthy chunk of change on your bill. For all such use cases you should of course benchmark a lot and then pick what fits.

Provided that you don't run into the issue of one end only talking in language X, which does not have JCOF support due to there not being any libraries for it yet. In the GitHub repo, it seems like there is only the JavaScript reference implementation for now.

The design documents are really thorough. I don't think I would be more than a day making a serializer in Python for example. Lack of libraries just means an interested developer needs to tend to the matter and solve the issue.

See https://news.ycombinator.com/item?id=32111239 where I tested gzip vs jcof vs jcof+gzip (tl;dr: you save 2M at the cost of an extra CPU intensive process)

You just blew my mind that “mutex” is a portmanteau of “mutually exclusive.” How have I been programming for 20 years and never realized this?

Perhaps you haven't had to program concurrencies or thought much about bitwise operators. It's not very frequent anymore. Used to be.

People keep trying to re-invent a human-readable, parsable, efficient, robust format that can be validated.

But "easy" wins so we keep returning to CSV and JSON. ^_^

Gzip may be standard and supported transparently by your http library.

OTOH zstd [1] should be significantly more efficient, while being as fast or faster.

[1]: https://en.wikipedia.org/wiki/Zstd

Reply on news.ycombinator.com