A practical guide to protocol buffers in Go | Hacker News Reader

A practical guide to protocol buffers in Go

minaandrawos.com

40 pointsnallerooth11 comments

The current golang protobuf api / implementation is unsuitable for large scale system. See https://groups.google.com/forum/#!topic/Golang-nuts/vxJNXdiM... for discussion (you have to scroll down the thread a bit).

That thread points out a lot of problems with both the current implementation of protos in Go and the inherent difficulties mapping of a code generator that was designed for C++ onto Go. I have had to deal with those issues and more using protos in Go.

However, it does not make them unsuitable. We have many large scale systems at Google written in Go using this proto library. Some are high QPS, some use large protos, some have complex definitions. Each requires care and thought, but they work.

Whenever you get a chance, would you care to share some of the main pain points to look out for from your experience as well as the workarounds? I haven't encountered major issues (small issues yes, major issues not yet) so far implementing protos , however my projects scales are typically not as massive as Google's. So for the sake of my quest to expand my horizons, any feedback will be greatly appreciated

The most important piece of advice is not proto specific: use pprof. It may sound obvious, but even though I have been using it for years, I find myself still not running it enough.

Proto decoding specific: if you're parsing a message with lots of fields, you can define a new message with just the fields you care about. I.e. if you're parsing

    message M {
      optional int32 F1 = 1;
      // ...
      optional int32 F100 = 100;
    }

but you only care about F1 and F2 in this message, define

    message SmallM {
      optional int32 F1 = 1;
      optional int32 F2 = 2;
    }

If you use SmallM to decode your messages, it will skip a lot of work filling out fields F3-F100, saving you a lot of CPU.

Lots of double exclamation points in there. I hate that I have a hard time taking it seriously because of such a trivial detail, but I do.

Yeah, it made me view the whole article as amateur and stopped reading.

It's a blog, not a peer reviewed ACM article. I read and learn from others no matter how they decide to express themselves as long as the content is relevant and helpful at solving current or future issues.

Thank you, I appreciate the comment

Interesting feedback, I didn't expect the exclamation marks to have such an affect. It is more of my personal writing style, however since there were two remarks on the exclamation marks, I edited the article and cleaned out some of the exclamation marks. Hope you enjoy the article.

I do find it an interesting article, I don't want you to think I'm entirely negative. I just kept stumbling at the punctuation. It's more about me than you.

Thoroughly interesting with just the right amount of detail to get off and experimenting on my own projects. Nice one!

Reply on news.ycombinator.com