Comment by cheald | Hacker News Reader

Comment by cheald | Hacker News Reader

JSON's appeal is that it is both compact and human readable/editable. If you're willing to sacrifice all semblance of readability/editability, then sure, let's all do the binary marshaling format dance like it's 1983.

Additionally, if you're sending data to a browser, then you're cutting the knees out of native JSON.parse implementations (Internet Explorer 8+, Firefox 3.1+, Safari 4+, Chrome 3+, and Opera 10.5+). The copy claims "half the parsing time" (just because of smaller data size), but I'm exceptionally skeptical of those claims since this is just going to move the parser back into Javascript.

I whipped up a quick example to demonstrate my point: https://gist.github.com/2905882

My results (using Chrome's console):

    JSON size: 386kb
    MsgPack size: 332kb
    Difference: -14%

    Encoding with JSON.stringify 4 ms
    Encoding with msgpack.pack 73 ms
    Difference: 18.25x slower

    Decoding with JSON.parse 4 ms
    Decoding with msgpack.unpack 13 ms 
    Difference: 3.25x slower

So MsgPack wins the pre-gzip size comparison by 14%, but then is nearly twenty times slower at encoding the data, and is over three times slower at decoding it.

Furthermore, once you add in gzip:

    out.json.gz: 4291 bytes
    out.mpak.gz: 4073 bytes

So, our grand savings is 218 bytes in exchange for 78ms slower parsing time. It'd still be faster to use JSON's native encoding/decoding facilities even if your client were on a 28.8k baud modem.

> I'm exceptionally skeptical of those claims

I can attest that these guys are known to post bogus benchmark numbers. The claim on their home page ("4x faster than Protocol Buffers in this test!") is incredibly misleading; their benchmark is set up in such a way that Protocol Buffers are copying the strings but Message Pack is just referencing them. It's true that the open-source release of Protocol Buffers doesn't have an easy way of referencing vs. copying, but it is still highly misleading not to mention that you're basically benchmarking memcpy().

One of these days I'll re-run their benchmark with my protobuf parser which can likewise just reference strings. I am pretty sure I will trounce their numbers.

I pointed out the Protocol Buffers comparison to a friend on IRC. He came back with:

"They rolled their own "high-level" serializer and deserializer for PB, built on top the lower-level stuff the documentation advises you not to use. Using the recommended interfaces, PB is faster than in their test. It is still slower than msgpack. Not sure why they'd make their test look cooked when they win in a fair test anyway. Further examination shows that the test is _mainly_ a test of std::string, for protobuf. std::string is required to use contiguous storage, whilst msgpack's test uses a msgpack-specific rope type."

I started reading their page and saw stuff like "take tiny, tiny data sample to have the most difference" and I started doubting. Since I saw the "faster" processing, and given how straight forward it is it parse json in C I was doubting even more.

So I clicked comments, expecting to see this, so that I wouldn't have to do it myself. Ah, makes me happy, thanks!

Wow, awesome seeing the benchmarks to back up the text of your reply. Thanks for doing that.

These results mirror the experiences I've had with large JSON payloads of time series data and experimenting with different wire formats.

The best message sizes I have obtained by custom compressors with complex predictive models and bit level encoding.

However with JavaScript on the critical path (say browser side) even a simple bit/byte bashing format like MessagePack was a performance loss and in my case the payloads were bigger.

YMMV, but if you really need aggressive compression or performance none of the standard solutions (protobuff et al) come close to hand rolled solutions.

If you have JavaScript on your path, I've had the most success by using JSON-to-JSON transformations utilising a predictive model (think say rows to columns and delta encoding) with gz over the top. This has got my payloads down in size without blowing out time - but it does spray the GC with lots of short lived object traffic.

Here's a conveniently executable fiddle if people want to exec cheald's benchmark themselves:

http://jsfiddle.net/javajosh/SjPub/

For the native JavaScript implementation you are correct - no doubt about that it is faster there, however, looking at the server-side it looks the other way around: for example parsing MsgPack in Ruby is about 5x faster than parsing JSON - you can rewrite your benchmark to Ruby and try for yourself

But you can install a c extension implementation for ruby (or pretty much any other language that doesn't already have native JSON built in) on your server fixing that performance gap. That's exactly what you can't do on the browser JS side for MsgPack.

then you could get a c extension for MsgPack, that would have to parse less bytes than the JSON c extension and therefore be faster

JSON is very good, no doubt about that - but i think that even JSON can be optimized, does not have to be by MsgPack, but probably by a binary format.

My point was that you can't install a c extension in a browser. I assumed that all the MsgPack libraries would be C or equivalent performance-wise.

As to text vs binary, I'll just leave this link here: http://catb.org/~esr/writings/taoup/html/textualitychapter.h...

i understood your point, however the speed in the user's browser is not important since it's just 1 operation, whereas the server has to do most of the operations decoding/encoding all that

i guess there are pros and cons on both sides, certainly nice would be something like a native BSON implementation with extensions in the browser/debugger to make it readable

Your blog post suggests using MsgPack as a drop in replacement for JSON for client-server communications. In that context every encode/decode operation on the server would be matched by one in the client browser. The <5% gain in performance on the server (and I think that is generous) cannot possibly make up for the performance hit of a javascript implementation of MsgPack.

When MsgPack was written a lot of JSON in the browser was still not native so an argument could be made if you desperately needed that bandwidth and it worked better than average for your specific workload. That is simply not true anymore.

i'm using it in Flash, so there is absolutely no JavaScript involved in my case...

>i understood your point, however the speed in the user's browser is not important since it's just 1 operation, whereas the server has to do most of the operations decoding/encoding all that

Speed in the user's browser is VERY important.

it is, but can you tell the difference between 0.5ms or 1ms? ;)

When you're doing it 100 times? Sure.

Reply on news.ycombinator.com