Comment by nly | Hacker News Reader

Comment by nly | Hacker News Reader

These things (Protobufs, Flatbuffers, Cereal, Cap'n Proto, Bond, Apache Avro, Thrift, MessagePack etc etc) are now a dime a dozen. In fact, even if you want a C++-only solution like this, you're still spoiled for choice (any of Boost.Hana, Boost.Fusion, Cereal, or Boost.Serialization for reflection + a little code for binding to your choice of codec). Hopefully one day we'll get standardized compile-time reflection as a language feature.

Imho, code generators and schema files are a feature. If you don't care about forward or backward compatibility, or portability to other languages, then just avoid pointers and memcpy your structs to disk. There are even libraries like Boost.Interprocess that will help you do just that... even with complex multi-indexed data structures like hash tables and maps inside mmap()'d blocks.

That all said, these days I use Flatbuffers, because it's super efficient, header-only, round-trips to JSON, the code gen is tolerable, and it doesn't lead to executable size bloat.

Code generators and schema files are not a good feature, IMO. One shouldn't have to duplicate everything in yet another "neutral" language and then maintain two copies of definitions.

The header-only libraries are good. Less dependency is good. No need for build system is good. Many of the existing solutions you mentioned have tons of dependencies. When code needs to be built reliably on 3 different OS and interfaced with arcane twisted build systems, dependencies become nightmare. Boost is great overall but I rather avoid this monster if I can.

You're using them wrong if you duplicate everything and maintain two copies. You should have one copy of the objects in the "neutral" language (protobuf or whatever) and compile the language-specific files out of that - but you don't maintain those any more than you maintain any other build output.

Code generators are good - if they are properly supported and tooling good. Have a look at the new work-in-progress about C++ meta classes (P0707): https://youtu.be/80BZxujhY38?t=3141 this stuff looks very neat.

> Imho, code generators and schema files are a feature.

Yes, especially if data has to be consumed by different languages. Personally, I don't see a need for yet another C++-only serialization framework.

It's insane that with this proliferation of serialization libraries none of them are still usable for real-world work.

Here in 2018, we still have to roll our own.

Libnop has the right idea, but still fails on two important points:

a) Versioning is a must-have feature. (And no, some weird non-standard 'table' type is not a replacement.)

b) The wire protocol needs to be human-readable ASCII. (Or at least a human-readable ASCII with no performance degradation option must be available.)

The table approach is fairly flexible. Both Protos and FlatBuffers do more or less functionally similar things. What specific deficiency do you see?

Take a look at the binary format spec:

https://github.com/google/libnop/blob/master/docs/format.md

The format is designed specifically to provide enough structural information that the binary format can be parsed without knowing the original structure definitions. This makes it easy to write a binary-to-string converter than can grok any payload with minimal complexity. Message observability was a primary goal during development.

Proto/Flatbuffers are okay for message passing, but many times you don't want to pass messages, you want to serialize C++ structures. There's no way to represent your 1Gb in-memory hashtable as a Flatbuffer, and that is not okay.

Writing a binary-to-string converter sucks. You write code once, and debug it every day. Anything that makes debugging harder is humongous pain that shouldn't exist; binary-to-string converters can be written, but let's be real for a moment: this is something that is dead last on the list of business priorities. If your message can be parsed with ad-hoc shell or Python scripts, that's a huge win for everyone.

I can assure you that Protobufs and Thrift are used in the real world.

Reply on news.ycombinator.com