Arguing against using protobuffers

I'm an actual author of Protocol Buffers :)

I think Sandy's analysis would benefit from considering why Protocol Buffers behave the way they do rather than outright attacking the design because it doesn't appear to make sense from a PL-centric perspective. As with all software systems, there are a number of competing constraints that have been weighed that have led to compromises.

- D

P.S. I also don't believe the personal attacks to be warranted or productive.

The utter awkwardness of Protobuf-generated code is particularly problematic. I've had pretty good results with the TypeScript code generator, but the Go generator is just egregiously bad. There's no way to write a client or server that uses the Protobuf structs directly as first-class data types without ending up looking like a ball of spaghetti, at least not if you venture into the realm of oneofs, timestamps or the "well known types" set of types.

I think we're at a juncture where we really desperately need a better, unified way to express schemas and pass type-safe data over the network that flows through these schemas. I'm currently working on a project that involves both gRPC (which uses Protobuf), GraphQL and JSON Schema, with the backends written in Go, and the frontend in TypeScript, and the overlap between all of these is ridiculous. TypeScript is a fresh breath of air that mostly solves the brittleness of JavaScript's type system, and it's absolutely crucial to extend this type-safety all the way to the backend.

The challenge is that no language is the same, and a common schema format ends up targeting a lowest-common denominator. For example, GraphQL errs (in my opinion, wrongly) on the side of simplicity and doesn't support maps, so now you have a whole layer that needs to either custom types (JSON as an escape hatch, basically) or emulate maps using arrays of pairs, neither of which is ideal.

Your "oneoff" and "repeated" might've been (I dunno really), a decision based on metrics where encoding repeated oneoff would've cost more, or made problems, more testing, or who knows what.

For one, Google was, and I believe still is open to discussing all kinds of matters, and you can get on to a design doc and comment if you like, if you care, if you have something to say... You can definitely influence, and people really did talk (you can even experience this from their public design docs - like from chromium, etc.).

And I love protobufs, so I guess I'm biased...

What a pain! Well, at least Google won't make that mistake again!

"Unlike most companies in the tech space, paying engineers is one of Google's smallest expenses."

According to here https://www.quora.com/How-many-software-engineers-does-Googl... there are ~28,000 engineers in 2014, which paying at 120,00 a year (glassdoor) would be... $3.3 billion a year? And this was 2014, and doesn't include stock compensation

Don't get me wrong - there are definitely problems with protobufs. But this post is citing a lot of stuff to back itself up that doesn't add up.

Also, required vs optional - definitely only optional, there should be nothing required. So I flat out disagree with that one.

> Option 1 is clearly the "right" solution, but its untenable with protobuffers. The language isn't powerful enough to encode types that can perform double-duty as both wire and application formats. Which means you'd need to write a completely separate datatype, evolve it synchronously with the protobuffer, and explicitly write serialization code between the two.

I'd say, get used to this. This is exactly the same story you have with ORM/ActiveRecord. You either treat it as a different layer and write the translation code that's needed, or mix it with internal logic and pay the price of messy codebase later. The problem isn't protobuff(or ORM)-specific. It's just a consequence of the fact that your internal business model and the data you want to save/send are usually two different things.

I come from an ONC/RPC background, which is the original UNIX RPC. Every iteration of an rpc would get versioned, and then you could write a conversion between versions, ex from V1 to V2, from V2 to V3, etc. This allowed for true backwards compatibility.

The idea of "forwards compatibility" is a pipedream, in my opinion. Just because something won't crash because a field is or isn't there doesn't mean that it will actually work. Code ages over time, and instead of looking at the spec, which will tell you which fields are required, you need to read the documentation, which may or may not tell you what is required. And especially when dealing with an older client, it simply may not work because what used to be optional-optional fields become required-optional fields.

It's the same argument as schemaless NOSQL database like Mongo vs MySQL. Having a schema is a pain in the ass, but it's better over time and ages much more gracefully than schemaless ones, because things won't just break.

Granted, the application I'm working on is fairly boring/vanilla so maybe I don't feel the pain points that come from going off the beaten path.

This is "worse is better" at work. protobufs are inconsistent and types map to Java. But it's a feature because Java is a language designed for for your average programmer. It avoids words like "coproduct types". Because most people's eyes will glaze over that and their mind will think "this is too complicated, there must be something simpler, like a map or something..."

> "Google is filled with the world's best engineers," and that "anything they build is, by definition, not built by amateurs.

That's true too though. It's a major factor when picking tools. So protobufs are used because of Google. gRPC is used because of Google. Kubernetes because of Google. Angular because of Google. Go became popular because of Google etc. Of course there are technical advantages to those things and they are great tools, but I think quite often the reason is simply ... because it's Google.

"nothing wants to inspect only some bits of a message and then forward it on unchanged"

In my experience it is extremely useful to partially parse a protobuf, or to not parse it at all and simply modify it by appending to it. Also useful is the ability to define a message type that is isomorphic on the wire with a different type, but is much cheaper to parse (example: a deeply nested structure where all of the fields of interest are defined at the top level).

Nobody even within Google will argue that no mistakes have been made with proto. Proto3 is generally acknowledged to have been a big mistake and it doesn't have a lot of traction within Google. Map and OneOf are both basically antagonistic to high performance application code, and people avoid those for that reason. But the author's arguments are pretty weak and his dismissal of Google itself as an existence proof is insufficiently supported.

One wonders if the author has practical experience with evolving a complex system composed of multiple independent services without downtime. Centralized sanity checking conflicts with transitioning services stepwise from e.g. foo.a foo.new_a by first deploying producers that create send both foo.a and foo.new_a and then phasing out foo.a after all consumers are updated.

Services which communicate via some general purpose, strongly typed schema language that's still lax enough to evolve backwards-compatibly are far easier to understand, maintain and evolve in my experience than alternatives such as json rest apis, overly tied down static or completely ad hoc protocols.

This article appears to be written by a programming language design theorist who, unfortunately, does not understand (or, perhaps, does not value) practical software engineering. Type theory is a lot of fun to think about, but being simple and elegant from a type theory perspective does not necessarily translate to real value in real systems. Protobuf has undoubtedly, empirically proven its real value in real systems, despite its admittedly large number of warts.

The main thing that the author of this article does not seem to understand -- and, indeed, many PL theorists seem to miss -- is that the main challenge in real-world software engineering is not writing code but changing code once it is written and deployed. In general, type systems can be both helpful and harmful when it comes to changing code -- type systems are invaluable for detecting problems introduced by a change, but an overly-rigid type system can be a hindrance if it means common types of changes are difficult to make.

This is especially true when it comes to protocols, because in a distributed system, you cannot update both sides of a protocol simultaneously. I have found that type theorists tend to promote "version negotiation" schemes where the two sides agree on one rigid protocol to follow, but this is extremely painful in practice: you end up needing to maintain parallel code paths, leading to ugly and hard-to-test code. Inevitably, developers are pushed towards hacks in order to avoid protocol changes, which makes things worse.

I don't have time to address all the author's points, so let me choose a few that I think are representative of the misunderstanding.

> Make all fields in a message required. This makes messages product types.

> Promote oneof fields to instead be standalone data types. These are coproduct types.

This seems to miss the point of optional fields. Optional fields are not primarily about nullability but about compatibility. Protobuf's single most important feature is the ability to add new fields over time while maintaining compatibility. This has proven -- in real practice, not in theory -- to be an extremely powerful way to allow protocol evolution. It allows developers to build new features with minimal work.

Real-world practice has also shown that quite often, fields that originally seemed to be "required" turn out to be optional over time, hence the "required considered harmful" manifesto. In practice, you want to declare all fields optional to give yourself maximum flexibility for change.

The author dismisses this later on:

> What protobuffers are is permissive. They manage to not shit the bed when receiving messages from the past or from the future because they make absolutely no promises about what your data will look like. Everything is optional! But if you need it anyway, protobuffers will happily cook up and serve you something that typechecks, regardless of whether or not it's meaningful.

In real world practice, the permissiveness of Protocol Buffers has proven to be a powerful way to allow for protocols to change over time.

Maybe there's an amazing type system idea out there that would be even better, but I don't know what it is. Certainly the usual proposals I see seem like steps backwards. I'd love to be proven wrong, but not on the basis of perceived elegance and simplicity, but rather in real-world use.

> oneof fields can't be repeated.

(background: A "oneof" is essentially a tagged union -- a "sum type" for type theorists. A "repeated field" is an array.)

Two things:

1. It's that way because the "oneof" pattern long-predates the "oneof" language construct. A "oneof" is actually syntax sugar for a bunch of "optional" fields where exactly one is expected to be filled in. Lots of protocols used this pattern before I added "oneof" to the language, and I wanted those protocols to be able to upgrade to the new construct without breaking compatibility.

You might argue that this is a side-effect of a system evolving over time rather than being designed, and you'd be right. However, there is no such thing as a successful system which was designed perfectly upfront. All successful systems become successful by evolving, and thus you will always see this kind of wart in anything that works well. You should want a system that thinks about its existing users when creating new features, because once you adopt it, you'll be an existing user.

2. You actually do not want a oneof field to be repeated!

Here's the problem: Say you have your repeated "oneof" representing an array of values where each value can be one of 10 different types. For a concrete example, let's say you're writing a parser and they represent tokens (number, identifier, string, operator, etc.).

Now, at some point later on, you realize there's some additional piece of data you want to attach to every element. In our example, it could be that you now want to record the original source location (line and column number) where the token appeared.

How do you make this change without breaking compatibility? Now you wish that you had defined your array as an array of messages, each containing a oneof, so that you could add a new field to that message. But because you didn't, you're probably stuck creating a parallel array to store your new field. That sucks.

In every single case where you might want a repeated oneof, you always want to wrap it in a message (product type), and then repeat that. That's exactly what you can do with the existing design.

The author's complaints about several other features have similar stories.

> One possible argument here is that protobuffers will hold onto any information present in a message that they don't understand. In principle this means that it's nondestructive to route a message through an intermediary that doesn't understand this version of its schema. Surely that's a win, isn't it?

> Granted, on paper it's a cool feature. But I've never once seen an application that will actually preserve that property.

OK, well, I've worked on lots of systems -- across three different companies -- where this feature is essential.

In my case, I have a single .proto file that serves as the input to two software systems: an embedded OS running on ESP8266 hardware, and the web interface served up by that OS to edit OS-specific features/variables/etc.

So, I define in one file, my types that I want shared both with the embedded OS, and with the web interface. On compile, both systems get their relevant generated code, and both systems are therefore able to communicate easily with each other. The user can use the embedded OS to change values coded in the protobuf, and the web interface that the system serves up immediately knows about the change. The web interface, by the way, generates the UI based on the protobuf declarations - meaning that adding a new var to the project is a simple matter of editing the .proto file, and doing a full clean build. Very, very nice: one change, multiple systems updated, and it produces an auto-generated UI for the new vars.

This kind of interoperability is very key - I'm sure there are alternative ways of doing this, but I can't think of anything as smooth and easy to integrate as protobufs have been .. maybe if we'd put a Lua VM in the mix somewhere, this'd be better, but .. protobufs are pretty tight.

Your constraints on protocols can change over time and context, changing something that is required to something that is optional can cause crashes in prod. Unless somehow the 'optional' design has a way to handle that? You also will have to do custom validation anyways, as your storage format will never be able to enforce all constraints you care about.

Also, it would seem like the 'debate' of optional vs required is harmful was resolved, as proto3 makes everything optional.

While oneof fields cannot be repeated, oneof fields can be arbitrary protos, so they can contain repeated fields. In other words, you can have a (pseudo-proto)

    oneof {
      RFoo {
        repeated Foo;
      }
      RBar {
        repeated Bar;
      }
    }

As for maps, I'd say don't use them. I have, I don't think they're very useful, except for prototyping things. `repeated (string key, string value)` is just as useful for prototyping things, and you should be quick to promote things to fields. Optional fields aren't costly.

I'm also in the minority who thinks that the `has_foo` methods are a smell, and everything should always be set to a default value[0]. If it really matters, one can define a maybe type via a oneof, but that should be the exception, not the norm. Most people disagree with that though, so maybe I'm crazy and you should ignore me.

[0]: But if that's not the case, everything should be optional, nothing required.

Yeah right, because insulting only Protocol buffer and Google engineering won't be edgy enough for a blog post... :/

This article was clearly written by amateurs.

The tone of this article is a little less than objective which I feel discredits it a bit. But hey it made it to the front page of hacker news so...

Now you lost one of the biggest benefits of using protobufs, the possibility to make things backwards/forwards compatible. When you add a field to the proto, a server using the new version of the proto will fail to deserialize a request from a client using an older version that's missing that field.

The author mentions this later, but doesn't really address it/propose a better solution, as far as I can see.

Protobufs looked promising because of their forward compatibility promise, but it looked like they only work if you write your code from the get-go using them. Rewiring a 15-year-old codebase would be a nightmare, so that was not an option.

Boost::serialization[2] looked promising, because it's non-intrusive -- but even "dumb" forward compatibility (skipping unknown fields) is not supported -- and reading its archives from other languages seems nontrivial.

So I went ahead and rolled out a custom framework with C++ template trickery which was non-intrusive, wrote to SQLite (with a fixed ORM schema), and did the job.

I wondered for a while if I was re-inventing the wheel, but in the end, I am glad I did it:

1. It wasn't that much work - especially compared to rewiring everything for Protobufs

2. It made me appreciate the goals and the API of Boost::serialization better - mine ended up being not too dissimilar. It was a great way to learn that - or any other- serialization API - without even looking at it! Now it's "let's see how they did ___" instead of "what does it do and why is this here".

Reading this article made me glad again: partly, for not depending on Protobuf -- and partly, for it having some of the same design flaws that I have (like having difficulty telling a zero-initialized value from a value not present in the data).

But the take-away for me was that sometimes, writing a custom framework is justified. Not all tools are suited for all purposes. Figuring out what one wants, and checking the list against the goals of a project can save a lot of headache.

[1] Binary was a necessity because some of the data is large arrays of full-precision floating point numbers, on the scale of 10-100MB. This was scientific data, no precision loss on saving/loading was acceptable.

[2]https://www.boost.org/doc/libs/1_67_0/libs/serialization/doc...

Protocol Buffers are not wrong, they simply have constraints, advantages, and disadvantages.

No language, binary format, text format, etc is free from advantages and disadvantages. All of them have different use cases.

If you are building a system where your data can be described by protobufs, it may be a good choice. If your data structures don’t mash up well and you have to manipulate them heavily, protobufs may be a bad choice.

Instead of pointing out use cases where a different serialization format may be better than protobufs and use cases where protobufs are better, and why, the author is spouting dogma about how protobufs are bad for every use case.

Be wary of working with developers who prefer to argue about why they hate certain technologies instead of providing useful data and ways to solve problems. You don’t always have to solve a problem just because you are aware of it but don’t go shouting from the rooftops that a technology sucks for every use case under the planet when that’s obviously not the case. The author’s opinion is more of: I don’t like protobufs. Not: protobufs are wrong

I recently saw a talk by Rich Hickey about Effective Programs [1]. The talk explains why Hickey favors dynamic types, with EDN [2] and transit [3] proposed as an alternative to more statically typed data exchange formats like protobufs and less structured ones like Json (the talk explains the reasons, nearing the end I think).

1: https://www.youtube.com/watch?v=2V1FtfBDsLU 2: https://github.com/edn-format/edn 3: https://github.com/cognitect/transit-format

This last is not the case. I've only ever seen it used when people a) already have written serialization code that works with XML/JSON, and b) don't want to write code to serialize to a binary format.

JSON also doesn't let you use objects as object keys, or have any form of polymorphism. It seems like the author wants protobufs to be an RPC system with a modern type system, when in fact it is a data interchange format.

>Of course, the actual serialization logic is allowed to do something smarter than pushing linked-lists across the network---after all, implementations and semantics don't need to align one-to-one.

I like that I can read a pb dump in hex and have it map quite closely to the protocol definition, rather than being viewed through heaping piles of abstractions. This is only going to make it more confusing and more work to use, with the only gain being to make some functional programmer feel squishy about their beautiful type system.

I largely agree with the "The Lie of Backwards- and Forwards-Compatibility" section. You can feed /dev/urandom into nearly any protobuf definition and it'll happily give you nonsense back, no errors. I understand why this is the case, and I'm not sure it should change, but it's bitten me several times.

[1] https://github.com/mgravell/protobuf-net

You get a statically typed Java interface, but underneath it a raw JSON object, and deserializing the proto is basically JSON.parse() + a cast.

e.g. something like this

MyProto foo = MessageDeserializer.deserialize(serializedProto); console.log(foo.getFieldFoo());

compiles to this JS:

var foo = JSON.parse(serializedProto); console.log(foo[1]);

I don't this complaint. You have to initialize data with something. proto2 had to "solve" this problem in C/C++/Go by making everything a pointer. How is dealing with null the "sane" case against zero-initialization?

What kind of systems have this architecture? Why would you forward on a message and not know its contents 100%?

You may not prefer it, but it works. Do something better, and make people use it.

But in the end, the author probably does not understand PB solves the inter-language data sharing problem, which makes all its complain secondary and inconsequential.

It's like complaining a car that drives fast and has great gas efficiency of cannot talk. (Well in 2018, it might be less of a stretch to demand a car that can talk)

   struct mytype s;
   s.field1 = something;
   s.field2 = something else;
   send(socket, &s, sizeof(s), 0)

   data MyData = MyData Int Int deriving Generic
   instance Storable MyData

   alloca $ \buf -> do
     let d = MyData field1 field2
     poke buf d
     send socket buf (sizeof d)