Hacker News Reader

Comment by meritt | Hacker News Reader

Sure, but if performance is the actual issue, then you'd likely pick an even faster and more compact wire format (proto, avro, thrift).

I guess the ability to have a schemalass and no type checking format (just like JSON), while enjoying some performance benefits? It just feels like a weird niche to me.

View on HN

Protobuf/Thrift require predefined schemas, which is a different model.

Avro is pretty similar to MsgPack in concept. It seems less popular (at least in my circles), and has fewer implementations, which may or may not matter. As for performance and efficiency, the first benchmark I found [1] shows MsgPack is more space efficient, and faster to serialize, while Avro is faster to de-serialize. The second benchmark I found [2] found exactly the opposite. So it's not clear to me that either is better, and I'm sure it depends on your data, which library you're using, and how you're using it.

Being nearly as performant as the predefined-schema serializers, while being nearly a drop-in replacement for JSON, seems like a major and valuable use case to me.

[1]: https://medium.com/@nitinpaliwal87/compression-and-serializa... [2]: https://github.com/saint1991/serialization-benchmark

> shows MsgPack is more space efficient

While I see the same numbers as you do, there's no way a format that includes plaintext keys alongside the values is going to be smaller than a format that only includes a packed field number+type and the corresponding values. There's something fishy going on here, and I can't seem to find a link to the source behind his benchmarks?

I'm not sure if the benchmarks above reflect this, but MessagePack allows you to use integers (or any data type) for map keys. This makes it a lot smaller than using plaintext keys, and makes it comparable to formats like Protobuf despite the lack of formal schema. (You also lose a lot of the context-free readability of messages, so it's a tradeoff.)

I guess. . . for my part, it's always seemed to me like messagepack falls into a sort of uncanny valley of serialization formats. Schemaless is very undesirable for internal APIs, IMO. Plaintext is desirable for public-facing APIs that don't need to be particularly performant, but only if you're willing to go full HATEOAS. If you're going binary for performance or don't want to bother with HATEOAS, I'm back to preferring a published schema over something that's (semi-)self-describing plus some documentation that's invariably incomplete or out of date.

I have never seen any benefits of HATEOAS materialise - it is too free-form to be machine parceable, is it far more difficult to navigate than Swagger or GraphQ. Maybe I dont get something?

I don't ever machine parse it, but it can be useful for manual discovery if you get an API that isn't super well documented. I think I might generally prefer Swagger, too, but not everything is implemented in a language that has a library like that.

At the end of the day, though, I'd seriously much rather just have a halfway well commented *.proto file. Work smarter, not harder. It's just that I won't recommend that for public-facing APIs because GRPC isn't universally well supported, while JSON over HTTP is.

Whatever was wrong with ASN.1 for the wire? it's not a direct replacement but a sanitising schema that's truly amazing in application breadth, support, tooling, and implementation efficiency.

In force documentation:

https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-X.68...

Blog on ASN.1 schema for JSON:

https://www.obj-sys.com/blog/?p=508

It's incredibly annoying to marshal and unmarshal arbitrary ASN.1/BER. Variable-length odd-bit-length length fields annoying. BER is like the worst parts of every other format, collected in a single standard.

I really like ASN.1.

The big difference is that messagepack is schemaless.

I don't know why but I constantly hear about things that never caught on and will never catch on in the future. Ok lets say I am hyped for ASN.1 as a schema for JSON. How do I integrate it into my bog standard Java application? Suddenly we are running into a huge problem.

>ASN.1 is a mature standard. As I already mentioned, it has been around since the 1980’s. Though stable, it is not stagnant; the most recent revision occurred in 2008.

It might be old but it is not widely used. To me it doesn't matter if something was created yesterday or 20 years ago as long as I can use it and using ASN.1 is significantly harder than necessary.

>ASN.1 has tool support. There are both commercial and open source tools that will generate code from your ASN.1 specification.

Ok, so where is it and why is it so difficult to find?

Even if we assume that we should just build it today then all the above claims suddenly become worthless. If someone builds ASN.1 tooling in 2020 then it is just as immature as e.g. GraphQL tooling that was built 2020. If there is renewed interest in ASN.1 then it might gain new features that will cause it to become less mature/stable again.

Having "mature" software is useless if it doesn't meet user demands.

There is an excellent article about ASN.1 and how it relates to JSON and other forms of encoding.

Here: https://www.thanassis.space/asn1.html

Sadly, ASN.1 is widely used — both TLS and SNMP use it, which means that we're using ASN.1 every time we read or post on HN.

ASN.1 is enormously more complex than JSON or even MsgPack, very bug-prone, has much worse tooling, is much less widely supported, and, surprisingly, most implementations are not even that efficient. Basically you pay a lot more in bugs and get a lot less in functionality.

I've spent a lot of hours poring over hex dumps of BER messages and CBOR messages (which are basically MsgPack) and I vastly prefer CBOR. But I prefer JSON way more.

don't forget https://capnproto.org/

It's just a nice default to pick. (But JSON is "simple" so msgpack did not really catch on. And probably won't for the reasons others mentioned.)

> It's just a nice default to pick.

Why would it be your default?

It is relatively efficient, schemaless, and cross-language and -platform. It supports floats and binary values as well.

It's my go to serialization format for redis payloads. Fast, compact, can represent a lot of types without too much magic.

It's really very-very fast, and no need for external schema registry. And it's really as simple as JSON. Has great language support.

https://github.com/thekvs/cpp-serializers#results

That said, currently I don't work with it in any projects. Apparently it hasn't really caught on.

Because all your existing json will just work, you don't have to change your data representation to support some other format(proto, avro, thrift).

Reply on news.ycombinator.com