28 karma · joined June 25, 2015
We're working on a certified course that will should be available at https://quantumformalism.academy/courses. It will cover post-quantum cryptography as well.
Also please feel free to check out our GitHub repo and take a look at our project Grid Ops (Java) https://github.com/nanosai.
Look, why do you even want to use Protobuf? You can just use raw bytes... why even bother with Protobuf - when you can invent everything yourself?
It is pretty clear from the text that Protobuf itself has no way of distinguishing the end of one message and the beginning of the next. Yes, YOU can add that yourself if you want. Like you can add hundreds of other things yourself. But Protobuf doesn't have it.
And - if you really, really love Protobuf so much, let me tell you a secret: You can EASILY embed a Protobuf message inside an IAP message (an ION object field) and there you go... ION works as the message delimiter.
Regarding the "marketing vs. science" comment - We would like YOU to make a comparison of binary protocols that is perfect, solely based on online docs and no access to the core developers... not all docs are equally good, and not everything we have read (e.g.on stackoverflow etc.)seem to be correct.
We don't gain that much by "marketing" a free technology. The customers of VStack.co will be communicating with VStack.co via IAP, regardless of whether IAP catches on outside. We just thought we'd share the work with others.
Yes, we are sure you can build a lot of stuff around Protobuf, but with IAP we will build all that stuff for you.
Sending schema over the wire is not a good solution for anything else than point-to-point communication. An intermediate node would need every single schema transmitted along with every single messsage, or have another way to keep the schemas cached. That becomes complicated.
The Protobuf documentation says very clearly that you cannot see when one message ends and another begins. Then a protobuf message is not fully self describing. This might be easy to add, but it doesn't have it (according to Protobuf's own docs).
We have looked at Cap'n Proto - but late in the process where we had already looked at quite a lot of formats. From what I can see, Cap'n Proto is pretty much just a binary struct. That is pretty close to what we wanted to do with ION, except we wanted it to be compact on the wire too. We have seen that Cap'n Proto has a compaction mechanism, but we have not yet had time to analyze and compare it to ION's.Cap'n Proto with compaction would be very similar to ION - on a conceptual level.
However, we need to make space for some IAP specific fields coming later in the process (like cache references, column stores and more). Stuff that is IAP specific. That is why we chose to roll with our own encoding in the first place.
Regarding Copy and Reference, the support for them is still not very good (= not automatic). But imagine your service executes an SQL JOIN query, and in that result a lot of objects are repeated (e.g. same zip + city for a lot of objects). The Copy field can be use to include the zip + city fields just once, and after that refer to them later with a Copy field. That is shorter than including them again. These two fields still need some work to have full support, but we are working on it.
Right now ION is the most well-defined part of IAP. The network protocol itself is still not 100¤ finalized. But, now that we are close to being done with ION (we still have extra fields to add as extended types), we can move forward with the IAP core protocols and semantic protocols. If we do not define a standard semantic protocol for remote object synchronization, IAP will be designed so that you can plug in your own semantic protocol to meet that need.
http://tutorials.jenkov.com/iap/ion-vs-other-formats.html
Not every innovation is from scratch, or 100% unique. MessagePack was designed as a binary encoding of JSON. CBOR too. ION was designed to be a binary general purpose data format able to mimic both JSON, XML, CSV and raw bytes (files), plus the most commonly used data types.
Of course there will be similarities when we try to tackle the same problems, but we also believe we had added something new and useful to the mix.
https://developers.google.com/protocol-buffers/docs/techniqu...
If you cannot know where one message ends and the next begins without a schema, it is impossible to use Protobuf messages as network messages when messages need to be routed. This should be pretty straightforward logic. Also, they write that Protobuf is not a good strategy for message bigger than 1MB.
From what we have been able to gather from googling about it, it is because Protobuf encodes raw bytes as var-ints which is inefficient. We have not been able to confirm this - Protobuf is weak in this area of the specification.
Also, since Protobuf is not good at encoding raw bytes (their words, not ours) we find it unsuitable as a general purpose encoding for messages in a network protocol, where we expect a lot of raw bytes to be exchanged (files, video frames, VoIP frames etc.).
Regarding GZipped JSON, it is true that GZiped JSON is small. But, due to the CRIME and BREACH attacks it is not recommended to compress data sent over encrypted connections (TLS).
If you look at our performance benchmarks page you can see a list of serialized length comparisons. As you can see, as soon as you send a few objects in an ION table, the difference is big. More than what you normally can gain with GZip (except perhaps for String).http://tutorials.jenkov.com/iap/ion-performance-benchmarks.h...
Furthermore, GZip only helps with transfer time, and actually slows down parsing time. If you look at our performance benchmarks you will see that ION parsing time is a lot faster than JSON. Additionally, if you really, really want high speed you do not parse ION (or JSON) into Java objects. You process the data directly in its binary form. If you look at our read-and-use benchmark you can see just how big a speed difference that gives. ION is designed for being processed directly. JSON isn't as good for that purpose.
Finally, ION is designed for fast arbitrary hierarchical navigation. JSON is not.
Additionally, Protobuf is not good at encoding raw bytes - according to their own words.
Additionally, if a message does not include its length up front, you are just trading faster write time for slower read time. A node receiving dynamically sized data then has the problem of knowing how much data to allocate for the full message, plus the receiver has to inspect the data as it comes in to see where it ends. With an ION message the receiver knows within the first 4-5 bytes (typically) how big that message will be, and can thus copy the following bytes directly into the perfectly allocated memory area, without having to examine them any further. Since data is most often read more times than written (e.g. data written to a file), we felt that making a tradeoff that favors read speed over write speed made sense.
Second, an array of arrays could mean anything. You have not semantics telling whether the arrays are independent or if the first array is an array of columns for the following arrays. ION Tables add that semantic information.
Third, yes, we could encode everything as text or as raw bytes and leave it up to the user to make sense of it. But that is exactly what we are trying to avoid with ION. We want to give devs a decent standard data format to use, that doesn't require a lot of data encoding choices up front. The encoding options have been thought through already, and sensible choices already made which you can just follow.
Fourth, you can nest ION tables inside ION tables, and thus create a more compact representation of an object graph. Using JSON / CBOR you would need a lot of nested arrays inside arrays to emulate that. Possible, but not exactly pretty.
Specification: http://tutorials.jenkov.com/iap/index.html
Benchmarks: http://tutorials.jenkov.com/iap/ion-performance-benchmarks.h...
Tutorials: http://tutorials.jenkov.com/iap-tools-java/index.html
Here is also a recent Infoq.com article http://www.infoq.com/articles/IAP-Fast-HTTP-Alternative?utm_....