Hacker News Reader

Comment by supermatt | Hacker News Reader

I took the first example from http://www.json.org/example.html and msgpack makes it 304 bytes. a simple gzip on the json is 289 bytes. The larger examples are even more in gzips "favour"

I am not 100% sure why I would choose to use this - maybe for super tiny documents?

View on HN

It's also likely faster to parse; string and array sizes are statically known, and all parsing happens in order, so there's no backtracking. Want to parse a string in JSON? okay, this is a starting quote, so let's iterate down the rest of the document, looking for unescaped end quotes, and then copy that into a buffer. In msgpack, it's a single memcpy. Additionally, with number heavy messages, then it shines.

{"red": 127,"green": 28,"blue": 27,"x": -18,"y": 27,"z": 2008,"time": 1583862185,"dollars": 1298,"cents":12,"name":"tom",timeseries":[1,2,3,4,55,66,77,218,239,340,9009,80008,400004]}

gives 106 bytes with msgpack and 157 with gzip. Also, if you gzip the msgpack, for your message example, it's 248 bytes.

And you trust a message from an unknown source? You can't simply memcpy according to some length indicator, that's just not safe. You still have to parse and validate.

Genuine question - is it dangerous to memcpy X bytes that we know must be interpreted as, say, an integer?

Potentially. Most network protocols are big-endian while x86 is little-endian.

No. Everything is 0s and 1s after all. Take for eg. a byte. It has 8 bits and by permutating all the 0s and 1s you end up with all the possible values of a signed byte; all the numbers -128 to 127. So now, if you were to copy a byte from a random memory location that byte will just contain a permutation of 0s and 1s which when interpreted as a signed int, will simply contain a number between -128 and 127.

That's what I thought ...

https://capnproto.org serialisation scheme skips the decoding. Does that make it not safe?

No serizalization is safe

- https://docs.microsoft.com/en-us/security-updates/securitybu... - https://en.wikipedia.org/wiki/Billion_laughs_attack - https://en.wikipedia.org/wiki/Zip_bomb - ...

None of those are serialization schemes. XML can be used for serialization, but if you look at the whole ecosystem it is a Turing-complete complexity monster, so of course it isn't safe.

It depends on what constraints apply to the data. Any bit pattern could be used for an int, but to guarantee a UTF-8 string it would need to be validated.

As long as you know the length of the entire buffer, you just ensure that:

  current_addr + message_len - start_addr < buffer_len

Or am I missing something?

buffer_len could be larger than the message, copying some incorrect things into memory.

Similar to HeartBleed, where there wasn't validation on the heartbeat message, and the server would echo back buffer_len instead of just what was sent.

I believe author intended buffer_len to be the length of incoming buffer (size of HTTP payload, number of bytes read from file, length of the database entry, etc...). So the worst that can happen is that entire input message is consumed -- like a JS payload which missed closing quote.

I can think of a very contrived situation where this can be a problem, but in most cases this will be perfectly safe.

Invalid unicode sequences?

Obviously you need to check that the length doesn't go past the end of the message, but that's a trivial O(1) check. You don't have to scan the bytes of the string first to decide if they are safe to memcpy.

You might want to validate those byte sequences are valid character encodings.

You should be doing that with JSON as well, so this isn't a pro/con of either format.

That was obviously my point.

ie just because MessagePack is a binary format it doesn't mean you can skip the same string checks that JSON requires; which means parsing MessagePack strings is unlikely to be any faster than JSON strings (contrary to the suggestions others have implied with the "just memcpy" comments). It's just with JSON that validation is done as part of the parser (remember JSON only technically supports a subset of ASCII and any extended characters or unicode is encoded via escape codes) where as with MessagePack you'd need to do that validation as an additional step.

Integers, on the hand, might differ since JSON would need additional validation (again, backed into the parser) which MessagePack would not because MessagePack encodes the integers as binary integers where as JSON encodes them as ASCII values that would need converting back to binary integers.

(hint: read the message I'm replying to).

Many (most?) applications do not actually care whether a byte blob of text is structurally valid UTF-8. They are either passing it around as an opaque byte blob, or already applying much stricter application-specific validation. Validating UTF-8 automatically at the serialization layer is a huge waste of cycles, especially in a big distributed system.

On closed systems where you control both the input and output, then sure (Though I’d still recommend against that particular short cut because it’s an easy way for bugs to go undetected).

However if you’re accepting MessagePack encoded data from insecure systems (such as end users) then you absolutely should be validating your input somewhere along the pipeline and it’s usually better to do that early on.

Also it’s not generally the distributed systems you worry about when it comes to this specific degree of micro-optimisation (which is basically what this is). It’s the monolithic ones. Distributed architecture is meant to solve various problems (for example but not limited to, high availability, reduced geographical latency, single site but running on cheaper commodity hardware, etc) but often at the cost of CPU cycles. Whereas your monolithic infrastructures where you have fewer servers (such as Stack Overflows set up) would be greatly more dependant on reducing computational overhead where corners could be cut. However they’d also be significantly less likely to need networked RPCs via MessagePack anyway (simply due to the monolithic design of their architecture).

Keep fighting the good fight.

> Want to parse a string in JSON? okay, this is a starting quote, so let's iterate down the rest of the document, looking for unescaped end quotes, and then copy that into a buffer.

Yes, so what's the problem?

Meanwhile keep in mind that the JSON document was already passed in a HTTP request body, or that its trivial to put string length checks on a parser, or even nesting limit checks.

Delimited formats are, in general, slower to parse than formats where the record sizes are encoded in the message.

That said, not something I want to prematurely optimize for.

I don't think it's that premature. Making the (big) assumption that serialization format X has good stable support in your language, and has the same expressive capabilities as JSON, you should be able to use it as a drop in replacement (as long as you control both ends of the communication...). And doesn't a huge portion of what we do around here amount to "shuffle serialized data around as efficiently as we can"?

> Delimited formats are, in general, slower to parse than formats where the record sizes are encoded in the message.

I don't see your point at all. The state machine required to parse a JSON string essentially has only 2 states: current token is either a string character or a string delimiter. That's it. Adding a record size increases the number of states because now you not only need to track if a token is a string character but also if the character makes sense to be there within that state.

Moreover, when you parse a JSON string, once you hit the string's end delimiter you already are able to calculate the string size. Thus, if the string is short enough to fit the buffer then not only is the lexer simpler but it also requires the same memory allocations than strig formats with record sizes. If however a string doesn't fit the buffer then we are already in the territory of a ropes data structure in both cases, thus the number of memory allocations tend to be equivalent.

Why would you make a comparison by gzipping one and not the other? Running gzip on the msgpack example reduced it from 304 to 248 bytes.

Sure, but I think what he meant is that what the website is showing isn't the real gains.

Without GZIP:

JSON 583 bytes

MessagePack 304 bytes

52 %

With GZIP:

JSON (GZIP) 289 bytes

MessagePack (GZIP) 248 bytes

85%

So MessagePack lets you skip the CPU and memory usage needed to compress the JSON yet get similarly sized messages to compression JSON. The compression of the JSON Is a non-trivial extra load on a busy server.

Or you just use JSON with gzip and never think about this problem ever again. Choosing msgpack vs JSON is a micro optimization at this point. There are significantly better alternatives if you want to save CPU cycles.

Most likely you'll have GZIP enabled on all your HTTP request. Your bandwidth will be a bottleneck way before GZIP CPU load become one.

I just took a quick look and they do compress their queries, using Brotli, which is even more efficient than GZIP. They are behind Cloudflare, so it's probably from them.

I just tried with https://jsonplaceholder.typicode.com/todos and I get this:

Raw:

JSON: 24,311

MessagePack: 14,704

60 %

With GZIP:

JSON: 3,965

MessagePack: 4,063

102%

With Brotli:

JSON: 3,495

MessagePack: 3,704

106%

So essentially, MessagePack compressed is WORSE than JSON, and not compressed, is at least 3 times worse than compressed.

So you're saying MessagePack with no compression is way smaller than JSON. Which means with no load for compression it has a 40% bandwidth savings. With compression it's negligibly worse than compressed JSON. Seems like a back foot argument for MessagePack over JSON. That's to say nothing of the encode/decode efficiency differences.

In the case where your data can't be cached but CloudFlare proxies and compresses for you, MessagePack wins because your uncompressed connection to CF is 40% more bandwidth efficient so egress out of your app server for any traffic is reduced. In the case where CF can cache responses you get the same back end bandwidth win for a negligible (if any actual) amount of extra egress out of CF.

Use what you want but if you need a schemaless serialization format, MessagePack works well and is compact for "free".

A different way of looking at this is that MessagePack is a worse way to compress JSON than gzip or Brotli, at least under some circumstances.

Performance. Overhead of JSON encoding followed by gzip will be slower and more CPU expensive than msgpack.

Sure, but if performance is the actual issue, then you'd likely pick an even faster and more compact wire format (proto, avro, thrift).

I guess the ability to have a schemalass and no type checking format (just like JSON), while enjoying some performance benefits? It just feels like a weird niche to me.

It's just a nice default to pick. (But JSON is "simple" so msgpack did not really catch on. And probably won't for the reasons others mentioned.)

> It's just a nice default to pick.

Why would it be your default?

It is relatively efficient, schemaless, and cross-language and -platform. It supports floats and binary values as well.

It's my go to serialization format for redis payloads. Fast, compact, can represent a lot of types without too much magic.

It's really very-very fast, and no need for external schema registry. And it's really as simple as JSON. Has great language support.

https://github.com/thekvs/cpp-serializers#results

That said, currently I don't work with it in any projects. Apparently it hasn't really caught on.

don't forget https://capnproto.org/

I guess. . . for my part, it's always seemed to me like messagepack falls into a sort of uncanny valley of serialization formats. Schemaless is very undesirable for internal APIs, IMO. Plaintext is desirable for public-facing APIs that don't need to be particularly performant, but only if you're willing to go full HATEOAS. If you're going binary for performance or don't want to bother with HATEOAS, I'm back to preferring a published schema over something that's (semi-)self-describing plus some documentation that's invariably incomplete or out of date.

I have never seen any benefits of HATEOAS materialise - it is too free-form to be machine parceable, is it far more difficult to navigate than Swagger or GraphQ. Maybe I dont get something?

I don't ever machine parse it, but it can be useful for manual discovery if you get an API that isn't super well documented. I think I might generally prefer Swagger, too, but not everything is implemented in a language that has a library like that.

At the end of the day, though, I'd seriously much rather just have a halfway well commented *.proto file. Work smarter, not harder. It's just that I won't recommend that for public-facing APIs because GRPC isn't universally well supported, while JSON over HTTP is.

Protobuf/Thrift require predefined schemas, which is a different model.

Avro is pretty similar to MsgPack in concept. It seems less popular (at least in my circles), and has fewer implementations, which may or may not matter. As for performance and efficiency, the first benchmark I found [1] shows MsgPack is more space efficient, and faster to serialize, while Avro is faster to de-serialize. The second benchmark I found [2] found exactly the opposite. So it's not clear to me that either is better, and I'm sure it depends on your data, which library you're using, and how you're using it.

Being nearly as performant as the predefined-schema serializers, while being nearly a drop-in replacement for JSON, seems like a major and valuable use case to me.

[1]: https://medium.com/@nitinpaliwal87/compression-and-serializa... [2]: https://github.com/saint1991/serialization-benchmark

> shows MsgPack is more space efficient

While I see the same numbers as you do, there's no way a format that includes plaintext keys alongside the values is going to be smaller than a format that only includes a packed field number+type and the corresponding values. There's something fishy going on here, and I can't seem to find a link to the source behind his benchmarks?

I'm not sure if the benchmarks above reflect this, but MessagePack allows you to use integers (or any data type) for map keys. This makes it a lot smaller than using plaintext keys, and makes it comparable to formats like Protobuf despite the lack of formal schema. (You also lose a lot of the context-free readability of messages, so it's a tradeoff.)

Whatever was wrong with ASN.1 for the wire? it's not a direct replacement but a sanitising schema that's truly amazing in application breadth, support, tooling, and implementation efficiency.

In force documentation:

https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-X.68...

Blog on ASN.1 schema for JSON:

https://www.obj-sys.com/blog/?p=508

It's incredibly annoying to marshal and unmarshal arbitrary ASN.1/BER. Variable-length odd-bit-length length fields annoying. BER is like the worst parts of every other format, collected in a single standard.

I really like ASN.1.

The big difference is that messagepack is schemaless.

I don't know why but I constantly hear about things that never caught on and will never catch on in the future. Ok lets say I am hyped for ASN.1 as a schema for JSON. How do I integrate it into my bog standard Java application? Suddenly we are running into a huge problem.

>ASN.1 is a mature standard. As I already mentioned, it has been around since the 1980’s. Though stable, it is not stagnant; the most recent revision occurred in 2008.

It might be old but it is not widely used. To me it doesn't matter if something was created yesterday or 20 years ago as long as I can use it and using ASN.1 is significantly harder than necessary.

>ASN.1 has tool support. There are both commercial and open source tools that will generate code from your ASN.1 specification.

Ok, so where is it and why is it so difficult to find?

Even if we assume that we should just build it today then all the above claims suddenly become worthless. If someone builds ASN.1 tooling in 2020 then it is just as immature as e.g. GraphQL tooling that was built 2020. If there is renewed interest in ASN.1 then it might gain new features that will cause it to become less mature/stable again.

Having "mature" software is useless if it doesn't meet user demands.

There is an excellent article about ASN.1 and how it relates to JSON and other forms of encoding.

Here: https://www.thanassis.space/asn1.html

Sadly, ASN.1 is widely used — both TLS and SNMP use it, which means that we're using ASN.1 every time we read or post on HN.

ASN.1 is enormously more complex than JSON or even MsgPack, very bug-prone, has much worse tooling, is much less widely supported, and, surprisingly, most implementations are not even that efficient. Basically you pay a lot more in bugs and get a lot less in functionality.

I've spent a lot of hours poring over hex dumps of BER messages and CBOR messages (which are basically MsgPack) and I vastly prefer CBOR. But I prefer JSON way more.

Because all your existing json will just work, you don't have to change your data representation to support some other format(proto, avro, thrift).

> Overhead of JSON encoding followed by gzip will be slower

How much time would it take you to either

a) turn on gzip support on your HTTP server and keep using JSON

b) rip out your JSON stuff from controllers and whatnot and replace it with a custom serializer/deserializer that's far from an industry standard?

A decade ago I benchmarked and used msgpack in a Python system instead of json because it was so much faster that it halved my hardware requirements. I think thrift and protobuf and stuff were around then but I don’t recall how they compared.

Edit: found my post https://stackoverflow.com/questions/9884080/fastest-packing-...

I made a similar choice for a Ruby project, with msgpack over RabbitMQ.

I no longer have any connection to this repo, but the benchmarks (and a schema validator gem I wrote) are here: https://github.com/deseretbook/classy_hash/blob/master/READM...

I fail to see the use case too. If you want a compact wire format, there are numerous options (protobufs, avro, thrift, etc) and their size is achieved by storing the data in a compact binary format and the schema separately.

With MessagePack the schema is embedded alongside the data, exactly like JSON. So it's just binary instead of text encoding, which saves some space, but as you pointed out, standard text compression algorithms are going to likely perform similar or superior.

One particular advantage of msgpack is that it allows embedding binary data without binary-to-text overhead (~30 %). I don't really see msgpack as something that makes much sense to use inside a browser, though.

First piece of software I ever wrote was a base92 encoder. When I looked at b64 is was even worse than the 3:4 I had been told.

B64 as originally used (with an extra >2.6% for line splitting) is handily over 35%. As we use it now it’s still a bit over 33%, because of the == endings to detect truncation... 66% of the time.

Msgpack is a different encoding, not quite the same as compression. What happens if you do both?

I'm not sure this is a fair apples to apples comparison. Some thoughts:

1. msgpack + gz would be a more fair comparison to json + gz when comparing file size

2. have you run any time comparison? It would seem to me that msgpack gets pretty close to json + gz with far fewer resources, time

Yeah, that is a very fair point. I suppose if i can shave a few bytes per request, why not, and if the parsing is indeed faster, than all the better.

I havent had need to deal with such constraints personally, but maybe its something I should consider more. It doesnt need to be a 10x win to still be useful, and if its a simple import to use then i would likely be crazy not to consider it.

Do a speed test comparison as well. Serialization to msgpack is likely much quicker than json + gzip. This is for latency sensitive applications.

Gzip'ing the msgpack encoded example results in 238 bytes.

Nobody gzip their json over https, this is due to compression attacks.

Not every use case is affected by BREACH attack.

JSON is mostly used on the web, and with HTTPS, where compression is disabled.

gzip is very slow. I don't know why anyone would still use it today.

really? what do you suggest that is faster? Our shop has tested a number of compression formats (xv, bz2, gzip, etc) and gzip is good enough and faster than the others we tested.

Well yah, those are all ratio-tuned codecs that are very slow. LZMA beats bzip2 on both ratio and speed so you might as well forget about bzip2 forever. Zstd, snappy, LZ4, or even brotli are probably better choices than zlib for most people. Brotli has LZMA-like ratios at dramatically higher speeds.

The algorithms you name are all rather outdated.

Typical gzip decompression speed is somewhere in the 200-250 MB/s region, compression is much slower. LZ4 for example tends to compress at ~600-700 MB/s, and decompress at several GB/s. zstd is tweakable over a very wide range of ratio-speed trade-offs.

LZMA(2) (xz) is a rather troubled format and should not be used any more. bzip2 has always been slower than gzip with usually marginally better compression. It has been irrelevant for a long time.

> LZMA(2) (xv) is a rather troubled format and should not be used any more.

You mean xz. Not sure where you got the idea that it’s a troubled format, but if you’re talking about the infamous “Xz format inadequate for long-term archiving”, IMO that’s just bzip2 authors taking a dump on xz for no good reason, and fortunately for us it’s bzip2 that’s basically irrelevant today, not xz.

> It has been irrelevant for a long time.

Funny you say that, my company uses bz2 for compressing pretty much everything.

And many companies use fixed-width record formats for data exchange... what's your point exactly?

I guess that it's not irrelevant. Deprecated, old, etc sure, but irrelevant?

https://quixdb.github.io/squash-benchmark/

Brotli is better, zstd is faster, lz4 is often good enough and a lot faster.

When you say, "very slow" what do you mean?

gzip is fast relative to other compression algorithms and relative to internet speeds.

gzip is slower than an ideal gigabit network, but faster than 100mb.

https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-co...

"gzip vs bzip2 vs xz performace comparison" is like comparing which centenarian can limp faster, it might be entertaining for some people, but is generally not relevant.

> gzip is fast relative to other compression algorithms

gzip looks fast perhaps compared to "xz -9", but not to anything modern.

Your comment could benefit from pointing out the alternatives that outperform gzip.

Check out zstd

Gzip is old but still good. It’s not as good as Zstd nor cases where lz4 and ilk shine, but it’s not fair to claim its slow.

Gzip is quite close to the Pareto frontier, meaning it is a good trade off of time and space. See the charts at http://mattmahoney.net/dc/text.html (And read up on the Hutter Prize!)

Reply on news.ycombinator.com