This is going to sound sarcastic but it's not: Can we get back to just putting the members of C structures into network byte order and sending that over the wire in binary, à la 1995?
>Isn’t this all horribly insecure?
>No no no! To be clear, we’re NOT just casting a buffer pointer to a struct pointer and calling it a day.
Isn't this a direct contradiction to your claim? Or have I misunderstood them?
There's a couple subtle differences:
* The struct is allowed to be shorter than expected, in which case fields past the end are assumed to have their schema-defined default values. This is what allows you to add new fields over time while remaining forwards- and backwards-compatible.
* Pointers are in a non-native format. They are offset-based (rather than absolute) and contain some extra type information (such as the size of the target, needed for the previous point). Following a pointer requires validating it for security.
(Disclosure: I'm the author of Cap'n Proto.)
Anyway even if you do choose the struct pointer hack (which I do not see advocated here) it can be done relatively well albeit requiring language extensions and a bit of care. Pragmas and attributes to ensure zero padding and alignment between members. No pointer members. Checking sizes and offsets after a read (the hardest part).
Something like that has to be rigorously tested or proven to be free of buffer overflows. It's so easy to attack with malformed messages. Parsers for remote messages are a classic source of vulnerabilities. It's hard to test this, because it's a code generator.
This looks promising as an attack vector for a big system built on microservices. If you can find an exploit in this that lets you overwrite memory, and can break into some service of a set of microservices by other means, you can leverage that into a break-in of other services that thought their input was a trusted source.
The "zero overhead" claim goes away as soon as you send variable length items. Then there has to be some marshaling.
This is outdated, I should remove it. Cap'n Proto has been reviewed by multiple security experts, though not in a strictly formal setting. I trust it enough to rely on it for security in my own projects, but yeah, I am cautious about making promises to others...
> Something like that has to be rigorously tested or proven to be free of buffer overflows.
I've done a bunch of fuzz testing with AFL and by hand. I've also employed static analysis via template metaprogramming to catch some bugs. See:
https://capnproto.org/news/2015-03-02-security-advisory-and-...
(That was... almost five years ago.)
> The "zero overhead" claim goes away as soon as you send variable length items. Then there has to be some marshaling.
Space for messages is allocated in large blocks. The contents of the message are allocated sequentially in that space and constructed in-place. So once built, the message is already composed of a small number of contiguous memory segments (usually, one segment), which can then be written out easily. Or, if you're mmaping a file, you can have the blocks point directly into the memory-mapped space and avoid copying at all -- hence, zero-copy.
So no, there is no marshaling.
[1] http://preserve.mactech.com/articles/mactech/Vol.14/14.09/Re...
You'd still have to figure out some way to serialize nested messages. Note that you can have recursive message definitions.
I hope not (I know you are being sarcastic). We should use something that's trivial to implement correctly, as well as easy to read and to debug.
By comparison, JSON contracts are an absolute joy to work with. We still practice strong-typing on both sides of the wire (we control both ends), and have pretty much nothing to complain about. If you are concerned with space overhead w/ JSON, simply passing it through gzip can get you down to a very reasonable place for 99% of use cases. I understand that there are arguments to be made against JSON for extremely performance sensitive applications, but I would counter-argue that these are extremely rare in practice.
While zero copy is nice, you cannot make it work when using compression.
You can build large hierarchical structures of messages with lists contained therein. It's pretty much how .mov/.mp4/many, many media container formats work. The technique dates back to the Amiga days.
They're actually kind of performance heavy for no benefit.
I get about a 37% compression ratio with extremely fast decoding, like 10 million packets per second off an SSD.
It was better than snappy, gzip, and bz2 for the trade-off of compression time, decompression time and file size.
As for protobuf: flatbuffers, capn proto, HDF5, and plain C structs all deliver much, much faster decoding time. It's really not the best answer for any serialization at this point but it's still inexplicably popular.
Try gzipping the protobuf. Binary encoding and compression are different things which can be stacked. Gzipped protobuf should be smaller and faster than gzipped json in basically all cases.
Not really. Encoding/decoding protobufs is straightforward, but not nearly as simple as you’re suggesting.
Dumping a struct on a wire is just a wishful dream that turns into a nightmare as soon as you need to send that to a service written in another language or running on another architecture.
Don't get me wrong - there's plenty of insanity in protobufs. But trying to cover the same use-case will not create a simple protocol.
Python library is an absolute nightmare. Their tests used to catch Exception, and what they ended up testing was basically whether their test try to access nonexistant attributes.
The issue is that capnproto is relatively more complex, and as such is harder to implement well.
Some compilers conform to same ABI in same system or similar system and work almost exactly the same, so you may grow old thinking that's how it is until it's too late. I think gcc, clang and Intel work almost the same in Linux and OSX.
In newer C++ specs, you can also static assert that the struct is a POD type to statically ensure that there's no accidental vtable pointer.
This argument pops up every time someone mentions this and every time it's completely uninformed.
Explicit alignment doesn't suffer from this problem nearly as badly (yeah, you might have to add some padding but that's hardly the end of the world -- and if you have explicit padding fields you can reuse them in the future).
This is a nice side effect of network byte order being the opposite of the dominant cpu order, though obviously it was never intended.