* cap'n proto https://capnproto.org/
* flatbuffers https://google.github.io/flatbuffers/
* hdf5 (for machine learning/numerical analysis/finance) https://support.hdfgroup.org/HDF5/
You're just not going to beat these formats in serialization speed with anything (just mmap the file, use. Good luck beating it, certainly not with protobuf). Seriously, use hdf5 for machine learning and you can restart experiments and ... well they start. They don't spend the first minute or 2-3 reading their data back in.
Protobufs focus on:
* standard (meaning it's perhaps a bad standard but it's standard) (also meaning of course you don't get to choose formats, or perhaps I should say your company will lose a lot if it lets developers choose alternatives to protobuf anywhere) (this is where the frustration comes from)
* streaming support (meaning you can write out and read in protobufs without having to keep the whole thing in memory)
* extendable (including sort-of kind-of backwards-forward-compatibility. Meaning old code can read in a new version protobuf, change something and write out a proto that still has the fields it didn't understand)
* language agnostic
* composability (meaning cat protobuf1 protobuf2 > resultbuf means resultbuf is deserializable)
* an attempt to efficiently store integers (meaning it's integer encoding format is bloody complex, but "space efficient", except not quite so efficient it doesn't need compression at which point, why bother ?)
And in this protobuf will beat the above formats (except cap'n proto).
None of the alternatives to protobuf existed when it was invented though. The only alternative in existence was ASN.1. And it beat that, by a LOT.
Parquet is gaining a lot of traction lately for this use case.
Mainly, I just think that interchange formats should be boring, simple, and ideally not incredibly slow. Protocol buffers at least meets those bars, even if it doesn't meet the consistency bar that the article desires. The whole "stop being a hipster and just use X" meme comes to mind. Probably realistically X is JSON in this day and age.
The problem with JSON for internal formats is that there’s often only one consumer and producer, so documenting the format rarely happens. Later, when you want to reimplement one side, you learn that there is no “one place” where you parse the JSON, but that you hand bits and pieces of it to completely unrelated areas of code.
Figuring out these as hoc formats is nigh impossible, so you end up just looking at the data over the wire and writing code that parses what you see. Except then you have broken code that doesn’t parse the less-common variants that have extra fields, don’t have extra fields, sometimes have their fields as stringified ints instead of just binary ints, etc.
Virtually every time I’ve seen JSON used as an internal interchange format, reverse engineering that format a few years later.has become a massive, error-prone, tedious, and time-consuming task. Save yourself from this ahead of time and pick formats that require a predefined structure, like protobufs.
The one place where I don’t think dynamic types have their place is in internal interchange formats. These can live forever, and they can change subtly over time. It’s the living forever part that changes the calculus.
Don't forget integer type defined by implementation. Very easy to shoot yourself in the foot in when consumers/producers are not written in the same language, even more if it's a mix of dynamic and static languages.
Nothing beats a well defined and commented schema.
Presumably you would be okay if the care and rigor were taken to define a schema and its physical representation just happened to be JSON?
How many of those people exist?
Protobuf has disadvantages, but mapping types are something I frequently make use of. losing out on them is a big deal
That said, I think maps are a fine feature for a serialization format to have. But, I haven't gotten around to adding them in Cap'n Proto because I have yet to hit a use case where I wasn't happy with a list of key/value pairs.
It follows sorta the same reason that Clojure became a fairly widely used lisp – mapping types are extremely common in real world programs. Having them supported first class makes a lot of common tasks quicker to get through.
What does bother me is that in the above generic type, "key" and "value" must be pointer types (i.e. non-scalar). The reasons are not unreasonable but it's definitely obnoxious in practice.
When my team was deciding on a binary serialization and RPC communication format, we needed something that ran on a Cortext m3 with no malloc, and every major phone platform (including WP at the time) and desktop OS.
It came down to cap'n proto, and Protobufs.
Protobufs runs everywhere. It won.