Hacker News Reader

Comment by importantbrian | Hacker News Reader

Isn't this the point of things like protobuf and thrift? It seems like everyone beyond a certain scale sends data around in some binary format rather than JSON.

View on HN

Yes, I’m wondering if people have bad experiences with CBOR, thrift, flatbuffers, protobuf, messagepack, etc. There are a huge number. I think Protobuf might be the best choice across all languages at this point, maybe?

But that’s my question, if you choose a binary format, what are the issues to look out for?

Example, I work with Java, Rust, Go, and Python at work. I have specifically had issues with Avro (not my choice) which is really well supported in Java, but not much else.

I dislike protobuf for the reason that the generated code is often quite bad (in terms of the interface in the language you are using) and in external projects it's often a manual step (sometimes with additional headache of protoc2 vs protoc3) to just build the project. In terms of message specification it's quite good though.

This is the reason we opted for building our own code gen system instead of Thrift or Protobuf: they both generate really ugly code in the languages we wanted to use

Still early days, but we've been using CBOR instead of JSON lately at work for interfaces that have "settled" and it's been great. Means that you can shake out the early integration issues using human readable JSON, then just switch the ser/de once it's all playing nice.

Binary data support is pretty nice too for avoiding multipart request bodies.

Ah, I see what you were saying now. Yeah, I don't have direct experience with those binary formats so I can't say. I'm not an Avro fan either but there are other binary formats like parquet that are easier to work with and at least in my corner of the tech world seems to have become the de facto standard.

I would have suggested ProtoBuf too.

That said, there was a comment in their post about not wanting to tightly couple the Go and Rust since they weren't sure they could accurately represent their structures in both langs.

I don't know anything about this turbo tool, but it seems relatively low-throughput, and they intend to migrate it all to Rust anyway, so it's probably not a big deal.

That said; I'd have done this with binary comms and not JSON, ProtoBuf or not.

I'm using protobuf for exactly this use-case, but I'd say that it's clearly designed as a network format first, and makes trade-offs as such.

For example, it prioritizes backwards and forwards compatibility -- which is not a concern for IPC where you control both ends. So no real optional or required fields. Comparing structures (Go) is awkward and uses reflection. Structs embed a mutex and preserve bytes for unknown fields etc...

MessagePack seems to make a better set of tradeoffs by comparison.

Reply on news.ycombinator.com