Hacker News Reader

Comment by VStack | Hacker News Reader

We have never claimed that Protobuf could not be used for network applications. We have claimed that Protobuf is a bad choice for messages have have to be routable by intermediaries that do not know the schema of a Protobuf message. Where does one Protobuf message end and another begin?

Additionally, Protobuf is not good at encoding raw bytes - according to their own words.

View on HN

The quote from the comparison page linked to above reads:

> The fact that Protobuf is not self describing makes it unsuitable as a network protocol message format

Which is demonstrably untrue, as many companies (Google being the obvious example) have built huge networks on nothing but protobufs. It also is very close to saying that protobufs cannot be used for network applications.

No. That is not correct. You cannot route Protobuf messages. You can only do point-to-point communication with Protobuf. This makes Protobuf unsuited for systems that require routing of messages, like a P2P network where not all communication may be point-to-point but have to be routed through intermediate nodes that do not know the schema of the messages sent. Or, systems that communicate via message queues.

Also, since Protobuf is not good at encoding raw bytes (their words, not ours) we find it unsuitable as a general purpose encoding for messages in a network protocol, where we expect a lot of raw bytes to be exchanged (files, video frames, VoIP frames etc.).

Where is their words that Protobuf is not good at encoding raw bytes? I don't see that anywhere. There is a "bytes" type specifically for that purpose.

Please Read this page:

https://developers.google.com/protocol-buffers/docs/techniqu...

If you cannot know where one message ends and the next begins without a schema, it is impossible to use Protobuf messages as network messages when messages need to be routed. This should be pretty straightforward logic. Also, they write that Protobuf is not a good strategy for message bigger than 1MB.

From what we have been able to gather from googling about it, it is because Protobuf encodes raw bytes as var-ints which is inefficient. We have not been able to confirm this - Protobuf is weak in this area of the specification.

I've read that page, I don't see their words where they say Protobuf is not good at encoding raw bytes. You have not been able to confirm it because you didn't spend 10 minutes trying it (add a bytes field to a message and see what the resulting message size is, add it to the benchmarks). I got to that "No" in the feature comparison and realized that this comparison was more marketing than science.

There are lots of options for routing Protobuf messages which are pretty straightforward logic. Most common is probably writing the length of the message first before writing the message. You could also break up the message into two parts, a header used for routing and the message. You could use UDP as the transport mechanism if your messages are small enough.

Actually we have been told by another reader recently that Protobuf can in fact encode raw bytes efficiently. We will update the documentation.

Look, why do you even want to use Protobuf? You can just use raw bytes... why even bother with Protobuf - when you can invent everything yourself?

It is pretty clear from the text that Protobuf itself has no way of distinguishing the end of one message and the beginning of the next. Yes, YOU can add that yourself if you want. Like you can add hundreds of other things yourself. But Protobuf doesn't have it.

And - if you really, really love Protobuf so much, let me tell you a secret: You can EASILY embed a Protobuf message inside an IAP message (an ION object field) and there you go... ION works as the message delimiter.

Regarding the "marketing vs. science" comment - We would like YOU to make a comparison of binary protocols that is perfect, solely based on online docs and no access to the core developers... not all docs are equally good, and not everything we have read (e.g.on stackoverflow etc.)seem to be correct.

We don't gain that much by "marketing" a free technology. The customers of VStack.co will be communicating with VStack.co via IAP, regardless of whether IAP catches on outside. We just thought we'd share the work with others.

Yes, we are sure you can build a lot of stuff around Protobuf, but with IAP we will build all that stuff for you.

Also, it seems like a big leap from "unsuitable for systems that require routing of messages but for some reason don't know the schema of the messages being routed" to "unsuitable as a network protocol message format"

Reply on news.ycombinator.com