Libnop: C++ Native Object Protocols
github.com
github.com
Imho, code generators and schema files are a feature. If you don't care about forward or backward compatibility, or portability to other languages, then just avoid pointers and memcpy your structs to disk. There are even libraries like Boost.Interprocess that will help you do just that... even with complex multi-indexed data structures like hash tables and maps inside mmap()'d blocks.
That all said, these days I use Flatbuffers, because it's super efficient, header-only, round-trips to JSON, the code gen is tolerable, and it doesn't lead to executable size bloat.
The header-only libraries are good. Less dependency is good. No need for build system is good. Many of the existing solutions you mentioned have tons of dependencies. When code needs to be built reliably on 3 different OS and interfaced with arcane twisted build systems, dependencies become nightmare. Boost is great overall but I rather avoid this monster if I can.
Yes, especially if data has to be consumed by different languages. Personally, I don't see a need for yet another C++-only serialization framework.
Here in 2018, we still have to roll our own.
Libnop has the right idea, but still fails on two important points:
a) Versioning is a must-have feature. (And no, some weird non-standard 'table' type is not a replacement.)
b) The wire protocol needs to be human-readable ASCII. (Or at least a human-readable ASCII with no performance degradation option must be available.)
Take a look at the binary format spec:
https://github.com/google/libnop/blob/master/docs/format.md
The format is designed specifically to provide enough structural information that the binary format can be parsed without knowing the original structure definitions. This makes it easy to write a binary-to-string converter than can grok any payload with minimal complexity. Message observability was a primary goal during development.
Writing a binary-to-string converter sucks. You write code once, and debug it every day. Anything that makes debugging harder is humongous pain that shouldn't exist; binary-to-string converters can be written, but let's be real for a moment: this is something that is dead last on the list of business priorities. If your message can be parsed with ad-hoc shell or Python scripts, that's a huge win for everyone.
I assume the reason why many functions take non-const pointers as arguments (rather than non-const references) is because it follows the google style guide? https://google.github.io/styleguide/cppguide.html#Reference_...
Cheeky question: have you Googlers thought about revising this guideline? It seemed weird to me when I read it years ago and it seems to be getting more and more unconventional. When I see a pointer argument in most C++ code now I would assume that passing a null ptr is not an error, but I see quite a lot of the code in this library doesn't check for null pointers inside the function body and would explode if you passed one in.
This doesn't mean I don't sympathize with the justification in the style guide: "References can be confusing, as they have value syntax but pointer semantics". If C++ had a non-null, non-reassignable pointer it might do a lot of what references do and be clearer, particularly for generic code. I don't really know, but references are what we have, they suit indicating the expectation of non-nullity, and it seems to me that the benefit of clearly communicating that expectation gets you more of a benefit than the confusion around semantics takes away.
Non-nullable types are helpful for implying pre-conditions. However, readability at the call site (e.g. Foo(&bar) might mutate bar, whereas Foo(bar) should not) is still considered more valuable in a large scale codebase. Passing nullptr as a pointer argument is generally assumed to not be okay unless explicitly documented as permitted -- this is opposite the assumption that you stated.
There are places exceptions are made, where the use of pointers is deemed more confusing than non-const references (e.g. move-maybe semantics in very specific cases). Ultimately, most code follows the default style guide.
Besides, nullptr dereferences are pretty easy to diagnose in a library like this. And more often than not everywhere else too.
but does it make it any more readable ? if you use the pointer anywhere else and have it as a variable then suddenly you don't distinguish anymore between a pointer and a reference. It would frankly make more sense to have empty `#define in` and `#define out` macros and make a small clang plug-in that checks correct usage in your codebase - e.g.
int foo(int x, const foobar& my_foobar, boo& my_boo);
foo(x, in fb, out b); // ok
foo(x, out fb, out b); // compile error void Baz(Bar*);
void Baz(const Bar&);
void Foo(Bar* mutable_bar) {
Baz(*mutable_bar); // Not mutated.
}
void Foo(Bar* mutable_bar) {
Baz(mutable_bar); // Possibly mutated.
}
void Foo(const Bar& bar) {
Baz(bar); // Not mutated.
}
void Foo(const Bar& bar) {
Baz(&bar); // Compiler error.
}
- Does serialization/de-serialization require dynamic memory allocation? From what I've seen it looks like it doesn't.
- How are you handling endianness? floating point numbers?
- How complete/tested is the library? Are you aware of projects using it?
This library works well in an embedded context. I've personally used it on Cortex-M class micro controller firmware.
The serializer/deserializer does not require dynamic memory allocation. Whether or not dynamic allocation happens depends on how you use the library. If you avoid using data types that perform dynamic allocation (e.g. std::vector) and use (or write) Reader/Writer types that use static memory (e.g. nop::BufferReader/nop::BufferWriter) then you should be fine.
There are some nice tricks you can use to permit protocols that have convenient dynamic containers on the host and static versions on the embedded device:
https://github.com/google/libnop/blob/master/docs/getting-st... https://github.com/google/libnop/blob/master/docs/getting-st... https://github.com/google/libnop/blob/master/examples/shared...
Endianness is assumed to be little because the vast majority of hardware is little endian. There are utilities to convert here:
https://github.com/google/libnop/blob/master/include/nop/uti...
These are not currently used by the Reader and Writer types included with libnop for efficiency, but are available for you to use in your own Reader and Writer types if you really need it.
Floating point is a much stickier problem due to lack of standardization across hardware. The library does not address this automatically and just packs floating point types in machine order. However, the library provides tools to help address the issue. One approach that works well is to use a fixed point representation for the wire type -- a value wrapper type is convenient for automating this:
https://github.com/google/libnop/blob/master/docs/getting-st...
See the Fixed template in the example. This is especially handy if you have a micro controller without floating point support or you want to minimize the size of the payload at the cost of range and/or precision.
The library has a complete suite of tests and 97.9% line coverage according to GCOV, with particular attention to conditional and error paths.
We use the library for a few internal embedded prototypes. I have not tracked its usage outside of Google since its recent release.
Best, C
Do you have any evidence of this? I would be flabbergasted if a modern compiler couldn't optimize out a no-op endiannes conversion.
Moreover, endian conversion templates are unusually frustrating in the current C++ standard. I wish there were a better way, but one does not currently exist.
https://github.com/google/libnop/blob/master/examples/interf... https://github.com/google/libnop/blob/master/include/nop/rpc...
I haven't documented it yet since it's not fully baked, but it's quite functional nonetheless. I have a working prototype of RPC over USB between a host PC and a Cortex-M micro controller. The ability to define constexpr dispatch tables is very convenient in a micro controller environment.
https://github.com/google/libnop/blob/master/docs/getting-st...
Personally, I just write serialization functions using fwrite/fread/write/read, but I’ve used projects which depend on it.
Instead of relying on serialization of existing data structures I actually try to use more general data structures that already exist in a single span of memory.
Then serialization is just a matter of writing them out from start to finish, or allocating them in a memory mapped file.
* libnop uses a macro called NOP_STRUCTURE to create its key-value pairs.
* Cereal has a macro called CEREAL_NVP.
* Boost.Fusion has BOOST_FUSION_ADAPT_ASSOC_STRUCT
* Boost.Hana has BOOST_HANA_ADAPT_STRUCT
* Boost.Serialization has BOOST_SERIALIZATION_NVP
...all these things work the same way to get reflection
Don't forget that libnop also has NOP_EXTERNAL_STRUCTURE (and friends) to decouple the annotation from the structure definition. This is handy when you have a C ABI with C++ internal implementation. I don't recall seeing a similar facility in other libraries.
Why is that? Is it kind of like the other project that was here recently, where it was created at FB but now independently maintained and not a corporate sponsored project (anymore)? Or is it deprecated and no longer recommended?
(I used to work at Google and had a lot of side projects. That was before Google moved everything to GitHub, but they liked for me to mark the code as copyright Google but "not an official Google product". I was fine with this arrangement.)
I believe these are still worked on company time, in which case it is absolutely normal for Google to claim ownership.
(To be clear, I meant the good/evil thing to be tongue-in-cheek...)
That's... disturbing, and all the more reason to keep your work and private life strongly isolated. Suppose outside of work you write scripts for various things and distribute them online to friends and so forth, or blog posts, etc. Your employer should never be able to claim ownership of that.