The Protobuf Language Specification
buf.build
buf.build
This is certainly The Buf Language Specification, which is useful in itself. Specs are like contracts. If I were to build a tool to be compatible with Buf, I would definitely aim it to work with this spec.
The problem is that the Protobuf project simply didn't sign this contract. Whatever it says is, sorry for the choice of word, a bit pointless if I'm trying to build a tool compatible with Protobuf, specially around forward compatibility.
The industry does need better tooling around protobuf/efficient RPC, and being dependent on a single company (i.e. Google) is definitely not healthy. I hope you guys succeed in what your are trying to doing.
* The language itself is unlikely to change much given it's been public for so long. A non-official spec that captures the current implementation is probably going to survive for some time.
* There's no official spec (which I would prefer) for me to base my tool on. This spec is about my only choice. The more tools targeting this spec, the hardest would be for Google to break compatibility with it, reinforcing my previous point.
I will keep the parent comment for context, and I don't retract the fact that I think the title is misleading. Otherwise, great work!!
And, as you said, proto isn't in a great position to be making crazy changes anyway.
Even for those not using Buf, we expect this documentation to be of interest to the community as it describes a large number of facets of protoc that were previously undocumented (and required examining the source for protoc or playing around with test source code to see what it expects and what descriptors it generates).
If issues are found with this spec, it is true that we'll most likely have to revise the spec to match the compiler, not the other way around. But no software is perfect: some variations will be due to bugs in protoc, which can be fixed in the compiler to properly match the spec. Over time, we'd love to see an outcome where a formal specification is the source of truth.
For now, our commitment is to make (and keep) this spec as accurate as possible to describe the Protobuf language, not some Buf dialect.
It looks like the original is now gone, but a fork has taken over. Looking at some comments, fighting to get MRI to adopt it may have burnt out the people behind it.
I haven't dug into this in great detail yet, but the hard thing about the proto "spec" is that there isn't one, and protoc lets you do all kinds of crazy things that are really hard to model in languages like Antlr. There were some poor choices in protoc dating back to when proto1 was first created that have been carried forward. Having 20 years of proto definitions lets people come up with some crazy use cases.
Definitely interesting for this company to create an EBNF definition for protobuf.
The open-source version has a hand-written tokenizer and recursive descent parser that is not too difficult to translate to EBNF. You'll notice that the section on numeric literals is a little wonky, because the tokenizer does a check that is hard to describe in EBNF. But it isn't too bad.
Also, some of the constraints of the language are in prose in this spec because they are easier to enforce using a semantic validation pass, instead of trying to model purely with a CFG. (Optionality of the colon in the text format, used in message literals, comes to mind.)
There are some things that technically _could_ be handled in the grammar, but they would make the grammar much more cumbersome to read and understand. So those things are also extracted into prose.
> Definitely interesting for this company to create an EBNF definition for protobuf.
For what it's worth, Google has also published an EBNF definition (the subject blog post contains links to those specs). But they are incomplete and not entirely accurate, which is a non-trivial part of what led us to writing and publishing this spec.
Comment placement is basically allowed anywhere by protoc, but how to get those comments within a Descriptor object for a proto is not well defined (there are places where you can put comments that are not available within Descriptor). It provides leading/trailing comments, but there are many other cases that are missed today (like comments embedded within a list of items in an array). Maybe this is a mismatch between what protoc allows and what Descriptor presents, but it's definitely annoying.
> If this SourceCodeInfo represents a complete declaration, these are any > comments appearing before and after the declaration which appear to be > attached to the declaration.
https://github.com/protocolbuffers/protobuf/blob/v21.5/src/g...
After re-reading with this caveat in mind -- that comments are only preserved before and after complete declarations -- I realized that it does retain the comments that are expected.
Long story short: the descriptor is bad as an AST if you care about recovering the original source. The descriptor is lossy. I'd recommend using a real AST for use cases that want something non-lossy: https://pkg.go.dev/github.com/jhump/protoreflect/desc/protop...
1: https://github.com/jvolkman/intellij-protobuf-editor/blob/ma... 2: https://github.com/jvolkman/intellij-protobuf-editor/blob/ma...
Furthermore, Protobuf is an IDL, not a full-blown programming language. This makes it ideal for this use case, for describing APIs and data structures.
Gob-encoded data structures are described with Go. While Go is great for writing server-side business logic, it is not as well-suited as a description language for data that you need to share with non-Go systems.
It's unfortunate, and not surprising, that Google hasn't made a protobuf spec. I realize there are a ton of protobuf fans out there but, personally, it just feels like a massive, awkwardly maintained mess that I'm forced to live with for (mostly) their benefit.
A great outcome for the ecosystem would be that Google chooses to engage with the community before making language changes. And the best possible outcome would be that the authority is eventually inverted: a specification doc becomes the definitive source of truth on the language and `protoc` is updated to conform to it, instead of vice versa.
For the most part the "constraint tightening" thing doesn't affect the language specification, at least in my experience. For example, there have been some changes in protobuf that affect things like serialization order. A change like that can break brittle tests that do things like checking that a function produces some exact serialized string/byte sequence, but they don't affect the semantics of the language.
The original creator gave up on the idea [0] but it was immediately taken over by others and is still maintained [1]. In case of conflict, though, Matz (lead developer on MRI), not the specification, is the source of truth [2].
[0] https://github.com/rubinius/rubinius-website-archive/blob/87...
While `protoc` remains the source of truth, this spec captures the syntax and rules accepted and enforced by `protoc` in a far more detailed way than the official developer site.
Buf is founded by engineers who spent a LOT of time working with Protobufs outside of Google. I was always the one saying "please don't write your own .proto parser" but I am convinced Buf actually knows what they are doing here and probably have all the details right.
Our industry has a whole lot of tooling and infrastructure built around JSON, and almost every piece of it could be way better if it were operating with well-defined types instead, in the same way that TypeScript tooling benefits vs. JavaScript. Google has had an all-protobuf ecosystem internally for a long time, but much of it will never be released publicly. So that leaves someone like Buf to really build it out. I'm pretty interested to see where they're able to take it.
(Etc.)
I'm not sure what this is meant to achieve in reality. There is still only one implementation that defines what the language actually is, and that is Google's protoc.
I'm my experience working with and writing alternative parsers for the '.proto' language, I've found that time and again Google's documentation for the format is either woefully vague, or directly contradicts the actual implementation. I don't see the value in a third party "spec" if what I have to do in practice will always be "whatever Google did in protoc".
Maybe you mean something besides the wire format. In that case, good luck, because that shit ain't protobuf.
> Maybe you mean something besides the wire format. In that case, good luck, because that shit ain't protobuf.
Naming's hard :) Being really pedantic, I think even Google calls the schema description language "Protocol Buffers" and uses phrases like "the Protobuf binary format" or "the Protocol Buffer wire format" to refer to the wire format. Colloquially, it's never confused me to just use "Protobuf" for both.
You could be writing a linter, a formatter, an implementation of the Language Server Protocol, a compiler that's not protoc, a way to apply semantic patches to large numbers of Protobuf schemas, or any number of other useful tools. There's clearly at least some demand for tools like this - partial implementations of most of these exist, often with some corporate backing.
Unless you're implementing a Protobuf runtime (google.golang.org/protobuf in Go, upb for Python, etc.), your experience seems unusual to me - most developers I've encountered read and write the wire format using one of the existing runtimes.
That said, it does sound like a lot of fun - especially if it's in lisp!
From the article, it seems this is not true anymore?
> We've built the [new Buf proto] compiler within the buf CLI to accurately match protoc.
But we do hope that eventually the official definition of the language will be a proper specification, not a particular implementation. (And maybe this document could be the start of that shift.)
So while there are multiple implementations (Buf, Square/Wire, probably others), the protoc implementation is canon.
It took two minutes to find but now I wonder, is there something wrong with it…?
It's been several years, so I can't point you to specifics. But if you look at open source projects that parse '.proto' files and do anything interesting with the result, you'll probably find lots of comments to the effect of "docs say X, but we are doing Y because that's what protoc does".
To my mind I'd rather have something written in Go that I can pull in and version using `go.mod` instead of having to special case a single tool, as well.
The biggest omission in the existing docs was the specification of the language.
Plugins generally require a library for a particular implementation language, so content we write would likely have to focus on a single language and library (at least to start). Whereas a spec is more broadly useful, regardless of what implementation language one is using with Protobuf.
https://gitlab.com/wireshark/wireshark/-/blob/master/epan/pr...
I would rewrite this maybe to:
> we are making Google's internal problems into everyone's problems
There are benefits to an IDL in the abstract, but an IDL for everyone should be built with the benefit of hindsight looking at the lessons of protobuf, ion, thrift, etc. Not just baking Google's internal backwards compatibility obligations into a formal spec everyone should follow.
I think any time google takes an internal tool and flips the "open source" bit on it, it turns out to be a bad match for the rest of the world. When they instead take the time to build a new system that learns from the internal tool, like Kubernetes learned from Borg, I think the end result is significantly more valuable.
Are these Google’s internal problems? Or, what google-internal problems do protobufs solve that nobody else needs to care about?
Edit: to your edit, I find it hard to see a different way to do things.
Maybe not perfect, maybe there are use cases and patterns where they totally suck, but they exist and are already being used and they generally get the job done. While a part of me will always want to spend time optimizing and carefully crafting the perfect system from the bits up, there are more interesting things to build on top of your serialization and RPC frameworks so I don't see any real value in spending more time creating a 15th standard.
I think you're right. One example that I thought about the last time Buf was on the front page, but didn't post at the time, was the lesson that Google (and @kentonv with Cap'n Proto) took from Google's specific challenges with required fields in proto2. The Cap'n Proto FAQ [1] includes this:
> Imagine a production environment in which two servers, Alice and Bob, exchange messages through a message bus infrastructure running on a big corporate network. The message bus parses each message just to examine the envelope and decide how to route it, without paying attention to any other content. Often, messages from various applications are batched together and then split up again downstream.
> Now, at some point, Alice’s developers decide that one of the fields in a deeply-nested message commonly sent to Bob has become obsolete. To clean things up, they decide to remove it, so they change the field from “required” to “optional”. The developers aren’t idiots, so they realize that Bob needs to be updated as well. They make the changes to Bob, and just to be thorough they run an integration test with Alice and Bob running in a test environment. The test environment is always running the latest build of the message bus, but that’s irrelevant anyway because the message bus doesn’t actually care about message contents; it only does routing. Protocols are modified all the time without updating the message bus.
> Satisfied with their testing, the devs push a new version of Alice to prod. Immediately, everything breaks. And by “everything” I don’t just mean Alice and Bob. Completely unrelated servers are getting strange errors or failing to receive messages. The whole data center has ground to a halt and the sysadmins are running around with their hair on fire.
> What happened? Well, the message bus running in prod was still an older build from before the protocol change. And even though the message bus doesn’t care about message content, it does need to parse every message just to read the envelope. And the protobuf parser checks the entire message for missing required fields. So when Alice stopped sending that newly-optional field, the whole message failed to parse, envelope and all. And to make matters worse, any other messages that happened to be in the same batch also failed to parse, causing errors in seemingly-unrelated systems that share the bus.
How common is this type of message bus outside of a very big, very custom infrastructure like Google's? Also, I think the right lesson to take from this would be that the message bodies passing through the bus should be opaque; they should probably be defined as byte arrays in the schema for the message envelope. The message bus shouldn't parse or validate those bodies any more than an IP router is supposed to parse or validate the bodies of IP packets (leaving aside, for the moment, stateful packet inspection).
And so the power of Protobuf and Cap'n Proto schemas, particularly the power to validate messages which is supposed to be a primary advantage of strong static typing, is limited because of a specific issue with Google's infrastructure, from which I think they took the wrong lesson.
1: https://developers.google.com/protocol-buffers/docs/referenc...
2: https://stackoverflow.com/questions/31801257/why-required-an...
If something is truly required, you're better off writing the validation logic yourself, so that you can change it when your requirements change.
Validation logic is often tricky, or at least tedious and error-prone (in part due to its tediousness). I would much rather have a battle-tested piece of software generate validation code for me.
I don't really get this aversion to required fields. Some data is actually required, and this happens fairly often in APIs in my experience. And if that changes, then often you are fundamentally changing what the API does, so it should be a new API.
> Some data is actually required, and this happens fairly often in APIs in my experience. And if that changes, then often you are fundamentally changing what the API does, so it should be a new API.
I can think of some fabulous counterexamples to this that aren't really public, but suffice to say that there are lots of places in code with
msg.foo = "" // This field is required, but has been deprecated in
// favor of msg.bar for <efficiency/external change/time>
// reasons.
Cloudflare is a pretty big one. There are others, but you're forgetting the much more common and pernicious problem: a proto serialized in a database. The reader, writer, and database itself need to be synced across time as the database takes the shape of a message bus that stores the proto, potentially forever, so now to make something un-required, you have to first do an in place column type change on a database, as opposed to a 3-phase column migration, which is a safe thing everyone adores undertaking.
Just don't use required fields, they're bad. Learn from Google's mistakes!
> Also, I think the right lesson to take from this would be that the message bodies passing through the bus should be opaque
Proto has support for this (in varying forms, there's the `bytes` type and the `Any` type), but these are distinct from required, so if you want to keep required, you need to make sure that the [every] message bus wraps all clients in any, which has, if I recall, a nontrivial performance penalty.
In practice the use case I had in mind was the Google Search pipeline. Simplifying quite a bit, there is (or was, 15 years ago) a front-end server (mostly renders HTML), several back-end indexes (serving different corpora, like web, images, maps, etc.), and a middle part that fetches data from all the back-ends and serves them up to the front-end.
The middle part is where the problems would sometimes appear. A back-end index would be updated to stop filling in some field of a search result which had previously been required. They would dutifully update the front-end code that depended on the field before changing the back-end to stop sending it. But if they knew the middle part didn't use the field, then they often believed they didn't need to wait for the middle part to do a release -- forgetting that the `required` part is enforced everywhere.
The middle part wasn't a dumb message broker, it would actually inspect the data too, for a variety of reasons. Many ranking algorithms were implemented in the middle part and they might want to inspect pretty much any part of the information attached to search results. So your idea that the middle part should just treat all the "payload" as opaque wouldn't have worked -- it would have been hard to draw the line between what should be opaque and what shouldn't be, and impossible to keep it updated as requirements changed. The whole exercise would have just been a huge burden.
No, this situation is not at all unique to Google. It is, in fact, quite common in any arrangement that involves three or more independently-managed components.
> And so the power of Protobuf and Cap'n Proto schemas, particularly the power to validate messages which is supposed to be a primary advantage of strong static typing, is limited
I would argue that a much more important power of these systems is their ability to manage change. It's extremely hard to fully test what happens when two different versions of your code interact. Almost all automated test setups only test the current version of the code at HEAD; they rarely attempt to set up old versions of some services. Even if they do, how many combinations do you test?
The genius of Protobuf (which Cap'n Proto mostly copied) is that it's pretty easy to reason logically about how a change will interact with older versions of the code. You add a new field, you know old code will just ignore it, and new code receiving data from old code will fill in the default value. It's rare -- surprisingly rare, to be perfectly honest! -- that people can't reason through this in their head.
The problem with `required` was that the implications of changing it were not intuitive, because it's not intuitive that it had implications for servers that don't even inspect the field in question.
The lesson for me is that validation has to happen at the point of consumption. You should not try to validate your data early in the pipeline because your validation rules will likely become out-of-sync with what the consumer actually wants over time. The only way to maintain sync is for the consumer itself to decide whether the data meets its requirements. This makes a lot of theorists uncomfortable, but I've seen in play out in practice so many times...
I believe you, and I'm sorry I was quick to criticize. Thanks for taking time to clarify.
> But most of them are based on the incomplete specs from Google's developer site. None of them can correctly predict what source files protoc will actually accept or reject 100% of the time.
I'd like to think I got pretty close with the plugin that now ships with IntelliJ. It even supports the 65-bit integer literal [1] that protoc happens to accept for proto2-style float and double default values.
With this as a starting point, it'd be nice to fix some of the pecularities that arise from "implementation as spec", such as that literal value, and the fact that colon optionality in text format is based on value type, not syntax.
1: https://github.com/jvolkman/intellij-protobuf-editor/blob/6e...
https://github.com/protocolbuffers/protobuf/blob/main/src/go...
The parser consumes a "-", then consumes a number. The number can be any 64-bit unsigned integer. It is then converted to a double. Finally, if a "-" was seen earlier, it is negated. So by accident, it ends up allowing the range of a 65-bit signed integer.
The behavior for integer fields is different; compilation will fail with an out of range error.
I've run into this misconception so many times over the last decade. Protobuf is much less than an IDL (intentionally so). It's used to describe the data of an interface but is completely unopinionated about all other aspects of an IDL. GRPC is a great example of how to use ProtoBuf in an IDL, but it could be used for other categories of interfaces (object oriented, etc). People treating ProtoBuf as an IDL make the mistake of concentrating too much on the data format and not about (imho) more important aspects like pre and postconditions etc, that make an interface an interface.
I also like where Ion goes with it's symbol tables. They provide much of the space efficiency that Protobufs provide with schemas, but gives you much more freedom and flexibility. I'm a longtime fan of msgpack but it becomes inefficient if you transport field names with it. The Ion schema also seems nice as an alternative to Protobuf schemas, but able to describe arbitrary data constraints too.
So we've taken the initiative. No rent-seeking.
> This as a community project defeats the purpose of creating a definitive grammar/spec that protocolbuffers/protobuf holds itself to - without that component, having a community-written spec is no improvement over the current situation from what I can see, and in fact could create even more fragmentation and confusion, just my two cents.
It looks like the protobuf team created a repo intended for this (https://github.com/protocolbuffers/protobuf-grammar), perhaps the spec could be contributed there, including a review/blessing from the official team.
https://gist.github.com/Cloudef/7a69eba054f61ee63d6ee59262e1...
I also did this benchmark for cap'n'proto, flatbuffers, protobuf and raw structs. This is already years old so it might not be accurate anymore, but what surprised me most was how different flatbuffers was compared to raw struct, as the design sounds like it should be merely extension to what is essentially raw struct + making sure the data can be portably encoded and decoded.
The text at the top of the page says:
> This is a language specification reference for version 3 of the Protocol Buffers language (proto3). The syntax is specified using Extended Backus-Naur Form (EBNF) …
> This format is distinct from the format of text within a .proto schema.
1) What would you say is the best use case
2) and what unfortunate misusecases have you come across
The wide variety of client languages is really nice. I'm fairly certain that I can parse a proto in any language I'm ever going to use.
The binary wire format is fairly straightforward, and is pretty tight without using compression. Fields are byte-aligned, and if you wanted to generate a binary proto message by concatenating a few things together it isn't very hard. And then you can use your proto definitions in whatever language you want to parse it. You can even parse your proto definition into a proto (Google provides the proto proto definitions, I think I got that grammar right) and write tools that generate whatever code you want easily.
I think the text format is under-utilized. It's my go-to for configuration files. You create the proto definitions with whatever structures you want to structure your config settings, and then use the official parser to parse a text file. It supports comments (I'm throwing shade at you, JSON), and is simpler than YAML, while adding that structure. You can also use command line tools to validate the file as a pre-commit, or translate it into a binary format if you don't want to rely on the text format.
They are also useful for describing domain models. This isn't surprising since domain models usually find their way into RPC schemas (since RPCs will often query or define model data). But they can also be used in other cases, such as for persistence and structured logging.
Some misuse I have seen involves trying to make a protobuf model the _only_ representation of a domain model: it is almost inevitable that a physical model (a representation of your data in a SQL database, for example) will need to vary from a logical model, and trying to make a single protobuf representation serve double duty can be a source of problems. Making protobuf schemas conform to physical storage constraints often makes for worse abstraction. The model becomes de-normalized (which can make constraints and relationships harder to model/enforce), and can even leak details that consumers shouldn't know about or care about.
Another misuse is using it for data that never leaves a process -- a program's private, internal state. If a data structure never needs to be serialized (to persist or send to another process), then you're better off using native data structures in the implementation language (which generally have far greater flexibility/expressibility as well as better performance).
2) Serialization for storage or transit
Unless the spec has changed, there's a fun issue. In protobuf there's no notion of null. So for instance a null list and an empty list are serialized identically. This leads to a nightmare in any system where null can have meaning by itself. I was always confused by this decision since there would be an endless number of ways to represent null that would have near zero impact on size or performance. The only cost I can see is the abstract one of making the representation less "elegant."
There are a of course endless ways to overcome this issue, but I don't think serialization implementation details should ever be allowed to leak into the model of what's being serialized. Just begging for issues down the road.
Proto2 has `HasFoo` methods, which distinguish between empty (unset/null) and set explicitly to zero fields.
I don't like that proto3 made this choice, I think it was wrong, but there was some rationale behind it beyond just "elegance".
However, you might be surprised about the room for ambiguity. For example, there are several mistakes in the grammars on the official developer site. And there is no place on the developer site that, for example, clearly explains how option names are formulated and interpreted. Even the way that relative references are resolved is not thoroughly described; it is probably the most complicated part of the spec because coherence/consistency wasn't keenly considered when the reference implementation in protoc was devised (https://www.protobuf.com/docs/language-spec#relative-referen...).
So if you wanted to write a tool for the language, one that could correctly parse and understand a source file, without the details in this spec that tool will almost certainly be incorrect and disagree with how protoc parses and understands the same source.