Hacker News Reader

Comment by jeffparsons | Hacker News Reader

> As of today, Protobuf is now a fully-defined language:

(Etc.)

I'm not sure what this is meant to achieve in reality. There is still only one implementation that defines what the language actually is, and that is Google's protoc.

I'm my experience working with and writing alternative parsers for the '.proto' language, I've found that time and again Google's documentation for the format is either woefully vague, or directly contradicts the actual implementation. I don't see the value in a third party "spec" if what I have to do in practice will always be "whatever Google did in protoc".

View on HN

The intent of this spec is to actually put "whatever Google did in protoc" into a readable format, so you don't have to read the C++ code. The official docs fall short on providing much of the details that are included.

That sounds great, but what's the governance story? Are the authors of the spec document committing to keeping the document up to date here henceforth? Are the protoc maintainers committing to have these folks involved in project direction decisions?

As of right now, the former. That is currently required for our tools to remain functioning (https://buf.build). If/when changes are made to the language, we update our tools (and this spec) to continue to be accurate.

Protobuf isn't too complicated, I've found the wire format docs to be some of the best among the avro/msgpack/thrift/etc competitors.

Maybe you mean something besides the wire format. In that case, good luck, because that shit ain't protobuf.

The wire format is fairly straightforward if you've seen a few binary encodings. The language used to write the schemas isn't quite as simple and regular as you might hope, though.

> Maybe you mean something besides the wire format. In that case, good luck, because that shit ain't protobuf.

Naming's hard :) Being really pedantic, I think even Google calls the schema description language "Protocol Buffers" and uses phrases like "the Protobuf binary format" or "the Protocol Buffer wire format" to refer to the wire format. Colloquially, it's never confused me to just use "Protobuf" for both.

Except I've written thousands of lines of protobuf format handling that never, or only extremely distantly, touch a schema file. But there's no reason you'd ever do the inverse, pushing protobuf schemas around with no intent to handle the wire format. As an abstract data definition format it's exceptionally poor, it only makes sense if you also want to use the wire format (which is... better than poor, especially as the commodity ones go).

> But there's no reason you'd ever do the inverse, pushing protobuf schemas around with no intent to handle the wire format.

You could be writing a linter, a formatter, an implementation of the Language Server Protocol, a compiler that's not protoc, a way to apply semantic patches to large numbers of Protobuf schemas, or any number of other useful tools. There's clearly at least some demand for tools like this - partial implementations of most of these exist, often with some corporate backing.

Unless you're implementing a Protobuf runtime (google.golang.org/protobuf in Go, upb for Python, etc.), your experience seems unusual to me - most developers I've encountered read and write the wire format using one of the existing runtimes.

That said, it does sound like a lot of fun - especially if it's in lisp!

The wire format isn't great compared to these, particularly in its handling of nested messages. Or rather, non-handling, since you have to encode the sub message to a bytestring. That makes encoders a bit slower (or complex), and you can't look at a message and see the nesting without a schema. Not great.

Or super-great, because decoders can't fail on nested messages they don't parse. This is a perennial problem in the large corpus of JSON we parse at my day job, and it would be fairly awesome to treat all nested objects as thunks to be decoded on access instead of crashing when synchronizing a crapton of JSON-encoded application state.

Funny, I have the opposite view. If you have invalid data, you should fail as early and loudly as possible, not wait until it's decoded later and you have no idea where the faulty data comes from.

I have both views - if you're going to do something with the data yes please reject as soon as possible - but also it sure is nice to be able to make a dumb pipe for either efficiency or architecture reasons sometimes.

> There is still only one implementation that defines what the language actually is, and that is Google's protoc.

From the article, it seems this is not true anymore?

> We've built the [new Buf proto] compiler within the buf CLI to accurately match protoc.

Our intent with the compiler in Buf is to match protoc as perfectly as we can. We want to instill maximum confidence in our users that Buf is a trustworthy tool and a suitable replacement for protoc. And to do that, we need the behavior to match.

But we do hope that eventually the official definition of the language will be a proper specification, not a particular implementation. (And maybe this document could be the start of that shift.)

So while there are multiple implementations (Buf, Square/Wire, probably others), the protoc implementation is canon.

Have you thought of creating your own version of the protobuf language, sort of like GNU C? You could have an optional flag to enable it, which would allow you to create your own official specification.

There is a specification: https://developers.google.com/protocol-buffers/docs/referenc....

It took two minutes to find but now I wonder, is there something wrong with it…?

I've used the Google docs as a reference to build software that parses '.proto' files and then generate code based on what I found. The docs are... incomplete, ambiguous, and in some places directly contradict the behaviour of protoc. So they're okay as a guide for learning how to write '.proto' files by hand, but a specification they are not.

It's been several years, so I can't point you to specifics. But if you look at open source projects that parse '.proto' files and do anything interesting with the result, you'll probably find lots of comments to the effect of "docs say X, but we are doing Y because that's what protoc does".

Out of curiosity: why write proto language implementations rather than protoc plugins?

(I don't work at Buf, but happen to be able to answer this) - the post at [1] describes the rationale for wanting something different than the Google compiler.

To my mind I'd rather have something written in Go that I can pull in and version using `go.mod` instead of having to special case a single tool, as well.

[1]: https://docs.buf.build/reference/internal-compiler

A great question: We do plan to add content about the plugin protocol to this site. While documentation for plugins is light, it is easier to find and to get a working plugin than it is to find the information needed, for example, to write a tool that performs static analysis on a protobuf source files.

The biggest omission in the existing docs was the specification of the language.

Plugins generally require a library for a particular implementation language, so content we write would likely have to focus on a single language and library (at least to start). Whereas a spec is more broadly useful, regardless of what implementation language one is using with Protobuf.

So that you can analyze and troubleshoot protobuf network traffic? Wireshark has a protobuf parser that integrates with our dissection API:

https://gitlab.com/wireshark/wireshark/-/blob/master/epan/pr...

Hrmm, I don't see why you would need to think about proto files to do this. You can dissect protocol messages on the wire using the descriptor. In fact, I would say that would be a good improvement to the code you just showed me.

Reply on news.ycombinator.com