Hacker News Reader

> calling people amateurs without really substantiating is a huge smell IMO. The average Google engineer isn't a genius or particularly amazing by any stretch

Note that Jeff Dean and Sanjay Ghemawat -- the original creators of Protobuf -- aren't average Google engineers. They are literally the highest-ranked engineers at Google (Level 11, "Senior Fellow", a title assigned only to the two of them last I heard), and they basically invented MapReduce, BigTable, Spanner, and a variety of other foundational distributed systems technologies. Jeff now leads the AI division while Sanjay continues to focus on systems infrastructure.

So yeah, "amateurs".

(Disclosure: I wrote Protobuf v2, but it was just a fresh implementation of the same design.)

While I agree with alecbenzer's original point, I don't think that Dean and Sanjay created protobuf is a signal that they made the best decisions. Everyone has strengths and weaknesses even within the same job family.

I've encountered plenty of genius-level engineers when it comes to algorithms, architectures, and distributed system design that wrote GOD-AWFUL unreadable and unmaintainable code. I wouldn't trust them to design an API or a common framework optimized for usability.

In fact I almost wonder if intelligence is a hinderence in such cases. When you're too much like Cypher, and "don't even see the code", all code and all library choices feel equivalent.

Sure, but you didn't address any of the points in TFA.

Whether Dean and Ghemawat are high ranking or not, do the points stand? Is the design of protobuf solid?

See: https://news.ycombinator.com/item?id=21873418

It's possible to criticize one part of the article (calling people amateurs) without criticizing other parts of the article.

As a Googler who has seen gPRC and Protobufs continually rammed into places where they clearly don't belong (ahem, embedded systems), I actually have a lot of sympathy with this article. I wouldn't call the authors of protobufs amateurs, but I do think there's flaws there, many of them around the type system, as this article points out, but also a lot around the language APIs.

My biggest concern with protobuf though is that it ends becoming the proverbial "I have a hammer, now everything looks like a nail" scenario. Every Googler goes through orientation with protobufs and gRPC, and then they proceed to stamp it everywhere... including places it may not belong. And then take it with them when they leave Google.

I think the article tone is inflammatory but most of the points are solid.

> many of them around the type system, as this article points out, but also a lot around the language APIs.

I think there's a lot not to love about gRPC/protobufs (used them at Google and again now at a startup) but I don't feel like this did a good job highlighting those issues.

I wrote my own serialization library for embedded systems recently. It serializes data that can be defined and initialized natively in C. The intention is for data section only data, not heap data. So for example, you can have variable length arrays, but you must have a maximum length to match the preallocated C array declaration.

The serialized format mimicks JSON, except that it supports tables- arrays of structs. This saves space compared with actual JSON, since the column names only have to be given once. Also tables where the columns are primitive types can be loaded directly into common spreadsheet applications. This is useful for my specific application.

The application has a CLI that allows you to browse the data with xpath like expressions. You can set or get any field of the hierarchy (using the serialization format), and since the type is known it can parse and schema check the user input.

I use the serialized format for storing a copy of the data in flash memory, so that it can be restored on boot up.

The metadata with the type information is all marked as const. In embedded systems, this data is placed in flash memory instead of precious RAM.

Sounds cool, is it open source?

> sprinkled in between a bunch of stuff that I'm pretty sure isn't an actual problem for anyone

I'd push back at this. After starting to use protos (even at Google) these issues smack pretty much everybody right in the face.

Also worth calling out: protos have evolved a lot since they were originally created, and there are very few people that are actually aware of the deep, dark corners of protos (think extensions, JS, Android...). (Got a complaint about what Google's open sourced for protos? You don't even want to imagine some of the batshit crazy stuff we did with it before that...)

> extensions

As someone who has done extensive work with Protobuf extension fields in Python, I can only echo that the API is nightmarish.

Using the word "amateur" to mean "inexperienced" or "unskilled" is a smell, or, more precisely, an equivocation: It lumps not being paid to do something with doing that thing poorly, which definitely aren't the same thing.

>This was like 2 or 3 sentences worth of good points sprinkled in between a bunch of stuff that I'm pretty sure isn't an actual problem for anyone. If anything, the things pointed out are issues for people trying to write code that manipulates protos generically, which is not what most people spend their time writing and is probably exactly the wrong thing to optimize.

Handling a serialization scheme "generically" is the "wrong thing to optimize"?

Sounds like the #1 thing anybody would want from it...

Have you ever worked in a polyglot ecosystem with rapidly evolving schemas?

Tools like protobuf and thrift were designed to facilitate schema evolution since interfaces in these ecosystems evolve quickly and independently. Generics undermine this by creating strict dependencies on a few types, making it difficult to evolve a single type without breaking things.

Poorly implemented generics would undermine one of the design goals of this project. In addition, there aren't nearly as many opportunities for generics in an IDL as in a programming language, so what would the upside even be?

Obviously you need certain core pieces of infra that handle protos generically (like serializing and deserializing) but

a) total SLOC for that logic is much, much less than SLOC for code that works with _specific_ protocol buffers

b) "consumers" of protocol buffers as a tool/technology mostly don't worry about what's going on under the hood of the generic serialization, etc. code

So it will often make sense to to make that core generic logic even significantly more complicated if it means making the stuff that everyone has to write over and over again even a little bit easier.