Show HN: Serialize++ – Tiny Data Serialization Library for C++ | Hacker News Reader

Show HN: Serialize++ – Tiny Data Serialization Library for C++

github.com

37 pointsapankrat44 comments

I’m curious what problem this is trying to solve that hasn’t been already solved in many other ways. There are so many serialization libraries for C++. The API for this isn’t too far from what YAS offers:

  YAS_OBJECT_NVP("myobject"
            ,("a", aa)
            ,("b", bb)
        )

instead of:

  OBJECT_SPEC( foo )
        __f( aaa ),
        __f( bbb )
  END_OF_SPEC

YAS also aims to do it with as little fuss as possible, similar goals, yet YAS binary archives are endian independent, and there are other advantages.

I take it's this one - https://github.com/niXman/yas ?

I haven't seen it before, but there are bound to be similar solutions to the same problem. The library I posted is used internally for a number of our projects and it has evolved in a course of several years. It is not _trying_ to solve a problem, it _is_ solving a very specific need that our code had since the beginning. So if you are implying that this is a pointless duplication of some other existing solution, then you are off. There's more than one way to skin a cat.

Re: endianess - to each (project) their own. We don't need normalized byte ordering, so with all other things being equal yas-like library will be marginally slower. Other advantages - I don't see any that would benefit our case, but I'm sure there are some that may come handy in other contexts.

You can add https://github.com/USCiLab/cereal to your footnotes as it works in a similar way: you list the fields of your structure in a serialize function.

Done.

I want to store data as binary, and generally I would just lay everything in arrays, write the vector size as an int, the type id (arbitrary value) as a char, write data, done. I just have 2 function to do that, and it's enough.

If you have fat data, binary is a wise solution. Generally flattening data in array is not so complicated, and you gain speed. SQLite is also a good solution, although I'm not sure it's good to store things like arrays of vec2, pictures...

I tried protocol buffers once, I was really horrified by the size of the headers it required.

Some quick and dirty things to make this useful, put it all in one header and wrap your code in a namespace. No really viable as a dependency otherwise.

Under 500 lines, this probably barely qualifies as a library, but it still allows packing structs and containers to/from binary blobs with as little effort as possible.

It has virtually no dependencies, requires no pre-processing (like ProtoBufs do) and it should be easy to understand and extend. It knows how to handle basic types and std containers and then uses a couple of C++ features to coerce the compiler into auto-generating all needed methods for custom types.

All you need to do is to specify which struct/class members must be serialized and then call store() on an instance to produce a blob. To restore an instance - feed the blob into parse() and that's it.

This was originally written to implement an IPC protocol for a pair of cooperating processes, but it can be readily reused for quickly storing data on disk and other things.

Being only five hundred lines doesn't disqualify it from being a library.

But being impossible to safely integrate into other projects because of illegal names and polluting the global namespace does.

This is at best a rough proof of concept which could at some point be turned into a library.

Such a clever swipe!

Don't know where you are sourcing your definitions from, but, conventionally, being a library merely means that it's a reusable piece of code that does something well-defined.

It's not reusable. Ergo not a library.

Does this work if the two communicating processes use different versions of the struct?

Depends on the differences, but generally - no, of course, not.

It'd be useful if you mentioned this in your caveats list, since version skew is a common gotcha that some serialization technologies (e.g. protobuf) are designed explicitly to address.

OK, will do. I assumed it was obvious that schemas on both ends must match.

I would assume it's obvious that schemas on the two ends don't need to match because solving that problem is usually a basic requirement of a serialisation library. Of course they need to be compatible in some way though e.g. in protobuf, don't reuse old field numbers.

Depends on the project. Lots of cases when there's no need for schema versioning. For example, when two processes need to talk to each other and they both run from the same binary or share the exact same datamodel. Even if they aren't run from the same binary, it's often much simpler to just require both sides to use the same protocol revision.

Looks nice to use, at least for the happy path.

I'd also say it looks a lot like the non-intrusive pattern for the boost serialization library. That's my default tool for this problem just because I'm pretty much always pulling in a pile of boost libraries for any non-trivial C++ project anyway.

Does it handle endianness?

No. See caveat #4.

The code stems for the same-machine IPC library. No point in doing any byte order normalization. Trivial to add though if needed.

> The code stems for the same-machine IPC library. No point in doing any byte order normalization.

Some processors can run one process in one byte order, and another process in another byte order! For example Itanium and the UM.be bit which the user can set.

ARM is also ambi-endian. Its rare to encounter big-endian ARM in the wild, but it does happen. TI's RM57 and TMS570 look an awful lot like they have the exact same silicon in them, except that one of them is wired for ARMEB and the other is wired for ARMEL. its likely configured for one or the other by burning a wee bit of OTP at the factory.

Nice, didn't know that. Seems a bit unfair though they didn't cover middle-endian order.

Seriously though, how many machines are configured like that in the real world and what's the overlap with our target installation base?

No idea - but it's something to keep in mind if you have the assumption in your head that 'byte order is constant for a machine'.

It almost universally is though, no? I'd expect a processor core with inverted endianness to be excluded from the general use and reserved for select processes that require that feature.

I don’t think so - it’s just another processor flag and processes can switch it as needed. For example you could switch while reading data from a network connection and back again afterward.

It'd be interesting to see any real-world uses of this. Must be something rather exotic that could benefit from not doing bswap/s in bulk.

I used to be careful to use Network Byte Order (defined in some RFC as big-endian) for everything. But for the last several years I use native byte order, and I haven't been sorry yet. It's significantly faster and simpler for x86, and big-endian systems are very rare these days.

Note that names containing double underscores are reserved in C++ for the C++ implementation, which your library is not part of. Using such names can lead to all sorts of difficult to diagnose problems.

Noted. From my perspective it's the same class of issues as with using _t suffix for custom C types - also reserved, so technically not good, but trivial to fix if it ever causes any trouble.

PS. In practice, the use of underscores, both single and double, as a name prefix is wide-spread. From the use of _foo notation to pass arguments to functions, to using _bar() to designate member functions that should be called under some sort of lock, to using __xxx for macros - I mean, yes, all this doesn't align with the C++ standard, but it exists, it's actively used in live code and it also leads to the coding habits that are hard to change. Hence the use of __f() in this particular library.

> he use of underscores, both single and double, as a name prefix is wide-spread.

Leading single underscores are OK at class scope, so OK for member variables or functions. They are not OK at global scope, in either C++ or C.

And, yes it is wide-spread, as are many other dangerous bad practices. It's something that is easy not to do, and which has no worth, so why do it?

Because it's not a "dangerous bad practice".

There are examples of actually bad and dangerous language misuse, but this is not one of them. It's mostly pedantry. Like using KiB instead of KB.

This detail is just one part of the library's broader disregard for the scope of names. One of the most difficult parts of managing large projects is dividing up the global namespace in a way that is equitable to all authors.

This library has a number of naming problems: It claims names that are reserved for the compiler and runtime libraries. It puts names into the global namespace. Worse, it puts generic names like "parser" and "store" into the global namespace. It also claims an extremely broad category as its own project name (Serialization++? Seriously?).

It doesn't matter how clever your particular library is. In total, the impact of those naming problems makes your library disrespectful to the broader C++ community.

It is dangerous. If you (illegally) #define a reserved name in your own code and then include a standard header file that (legally) uses that name, the results can be almost impossible to diagnose. There is never any need to do it, so not doing it is not "pedantry", it is simple good practice. The rules about double-underscores exist for very good reasons.

> the results can be almost impossible to diagnose

Humor me - concote a case with a standard header using __f name in a way that will make a conflict with this particular __f() macro "almost impossible to diagnose". I can't think of any.

Following overly broad rules that go against established practices without any due need is exactly that - pedantry.

Look at it this way - if a language spec was serious about reserving the __ name prefix, it's not hard to enforce it at the compiler level. But it's been a while since this made it into a spec and yet it's not enforced. So whatever the good reasons were behind this bit of spec, they were not a strong enough of concern to warrant any practical enforcement.

I haven't looked into what your macro definition would do to it, but just running

    grep -r '\b__f\b' /usr/include/

on my system shows a bunch of uses of this symbol. Looks like it's generally used as a parameter, though I also see it as a field name in a union, e.g. in /usr/include/math.h:

    __header_always_inline int __inline_signbitf(float __x) {
       union { float __f; unsigned int __u; } __u;
       __u.__f = __x;
       return (int)(__u.__u >> 31);
    }

I wonder how the compiler likes it if you try to #include that header after you've defined a __f() macro?

If you look at a standard library implementation, they go to extreme pain to uglify their symbols not to conflict with the user namespace. If you encroach on the library namespace your code deserves to be broken.

> Like using KiB instead of KB.

They mean different things.

Outside of the context of hard drives, KB is almost always used to mean KiB

> trivial to fix if it ever causes any trouble

So your library exposes __foo or foo_t, a program depends on your library and some other library.

Now a new standard comes out and adds __foo or foo_t to the standard library, and the program upgrades to the new standard. So far everything works.

Now the other library upgrades to the new standard as well and uses standard __foo or foo_t. Now the program experiences weird linker errors or runtime undefined behavior.

Trivial to fix.

Single underscores outside of the global namespace are perfectly fine in C++ (_foo is a common naming notation for members).

Using double underscores in identifiers is dangerous as you risk colliding with implementation macros or builtin, but of course if you make the identifier long enough the risk is minimal.

But defining a double underscore single letter identifier and the using it for a macro is really inexcusable.

__f is a questionable name for a macro.

But it's a petty irrelevant detail. I can't fathom how anyone could seriously see this is as a subject worthy of a prolonged discussion. The repo is a demo for a serialization technique, yet only one comment out of 30 is on that. Can't say I'm shocked, but it is moderately disappointing.

Welcome to HN! That's what we like to do here.

In all fairness that macro is a pretty prominent part of the library interface.

(Also full disclosure, I've also used one letter macros in my c++ experiments and yes I'm moderately ashamed).

Heh, yeah. It wasn't this bad in 2008 when I first got here... though I suspect it's more of a language issue.

C++ tends to prompt a discussion of a poorer quality. Every second person is an senior expert that feels obliged to teach everyone else the true ways of the language. Hence all the focus on irrelevant minor fluff like here. Especially evident when compared to C threads, where nitpicks are of a far better quality and actually relevant.

It takes at least two to tango. It looks like you’re responding to this criticism very defensively. An initial simple “thanks for pointing that out” response could have probably avoided the whole thread.

C++ tends to bring out the language lawyers because it is such a minefield of undefined, implememtation-defines, and unspecified behavior.

Reply on news.ycombinator.com