Hacker News Reader

Comment by mgaunard | Hacker News Reader

No one has come up with a binary format that has the same properties and has been as successful.

Maybe because most binary formats can't be human-readable which is one of important properties JSON do have.

You can easily write a tool to dump a binary format.

Now embed that tool in all my other tools so I can human-read the binary format without thinking twice and then we can chat.

such as https://en.wikipedia.org/wiki/Protocol_Buffers ?

last time I read, simdjson still beat protocol buffer a bit.

Due to the inherent constraint of JSON, the exact use case would matter a lot for such comparisons. Simdjson is generally faster when you only want a well-formedness check or a very small proportion of a large input JSON, but the "well-formedness" for JSON would be a small subset of the "well-formedness" of binary formats, and a partial parsing performance is often dominated by language bindings instead of underlying parsers (it is a wise move that simdjson also has a JSON pointer support for this reason, because that greatly reduces FFI overhead). Binary formats in comparsion tend to have a generally flat performance profile.

It's not self-documenting, you can't decode it unless you have the .proto. JSON just works.

Yeah, that's a great example of a non-human-readable binary format

IFF was basically that on the Amiga:

https://en.wikipedia.org/wiki/Interchange_File_Format

Unfortunately with the Amiga itself being a niche platform, the idea didn't catch on.

It also sort-of requires a central registry for the top level FourCC codes (basically reserved magic number for each file format).

Possible future candidates:

https://capnproto.org/

https://arrow.apache.org/

Arrow is columnar storage, nothing relevant here.

And capnproto is basically another version of pb... (see discussion above)

Look again at Arrow. Part of the concept is the wire format is the same as the in memory format—no de/serialization. Arrow isn’t intended for storage; they point to parquet for that.

Apache Arrow defines an inter-process communication (IPC) mechanism to transfer a collection of Arrow columnar arrays (called a “record batch”). It can be used synchronously between processes using the Arrow “stream format”, or asynchronously by first persisting data on storage using the Arrow “file format”.

The Arrow IPC mechanism is based on the Arrow in-memory format, such that there is no translation necessary between the on-disk representation and the in-memory representation. [0]

The Arrow spec aligns columnar data in memory to minimize cache misses and take advantage of the latest SIMD (Single input multiple data) and GPU operations on modern processors. [1]

0. https://arrow.apache.org/faq/

1. https://arrow.apache.org/docs/js/

msgpack and "I can't believe it's not msgpack with the name changed to my own name" CBOR are basically that.

https://en.m.wikipedia.org/wiki/Smile_(data_interchange_form...

Reply on news.ycombinator.com