Apologies for being a debbie downer, but it looks like the authors looked into Protocol Buffers and Flatbuffers.

Did they consider CapnProto?

In my limited experience with efficient serdes for analytics, CapnProto would meet 1 - 4.

Arrow is an in-memory format and CapnProto would afford the ondisc (memmap) heavy lifting with (very) little overhead. My CapnProto/LMDB datastructures allow me to write better than C++ code in Python with comparable runtime performance.

The only issue I have with CapnProto is getting it to run on Windows 7 x64, but it works out of the box on Linux.