Hacker News Reader

Comment by mantraxB | Hacker News Reader

The problem with FlatBuffers and Cap'n Proto is that while in-memory rep matching serialization formats are much faster (infinitely faster as Cap'n Proto says tongue in cheek) for languages with unsafe direct memory access (like C/C++), they're actually slower and more cumbersome to use in any language without unsafe direct memory access (like Java, Rust, Python, Go, JavaScript/Node, Swift and... well, most languages).

The other problem is that Google tends to release random stuff as open source and then modify it (breaking BC) or abandon it without regard for the people who are using it.

So caveat emptor, I guess.

View on HN

Java has ByteBuffer, Javascript has TypedArrays, Python has the "struct" module, and other languages have other fine ways of reading raw data in this kind of format. Some languages may lack the ability to inline calls well enough for true zero-copy to be efficient, but the worst case is you fall back to parsing the message upfront like you would with any other encoding. That upfront parse is still going to be faster, because it's just easier to read fixed-width values than variable-width.

And FlatBuffers actually uses ByteBuffer as part of its Java implementation.

I don't get it, you can't read anything out of a ByteBuffer, except a bunch of bytes.

You can't map 1:1 any more complicated in-memory structure to that buffer, or am I wrong?

ByteBuffer has methods for reading primitive values of all sizes -- ints, floats, etc. -- from arbitrary offsets within the buffer. So you just need a wrapper object with nice generated methods for each field which in turn call the ByteBuffer getters.

None that the C++ code works this way too. We don't cleverly generate a struct that just happens to have the right layout; we generate a class that wraps a pointer and provides inline accessors that read/write values from the correct offset relative to that pointer. After inlining, it's just as fast.

Rust does have unsafe direct memory access and a work-in-progress port of Cap'n Proto (https://github.com/dwrensha/capnproto-rust).

If any Pittsburgh HNers are interested, dwrensha is likely to be giving a presentation regarding his Rust-based Cap'n Proto library sometime in the next few months.

http://www.meetup.com/Pittsburgh-Code-Supply/

Reply on news.ycombinator.com