Protobuf Editions are here: don't panic | Hacker News Reader

Protobuf Editions are here: don't panic

buf.build

31 pointsperezd55 comments

Oh look, Google is shipping their org chart again.

  > What problems are Editions solving?
  >
  > In short, nothing really (for end users). The introduction of Editions
  > is the result of major Google-internal refactoring of how protoc and its
  > plugin architecture implements and observes feature checks when generating
  > code. This isn’t intended to force breaking changes to existing projects,
  > nor is it designed to impact any of the existing encodings.
  > 
  > It should be a boring change that gives plugin maintainers finer-grained
  > control over how future versions of their Protobuf runtimes behave,
  > improvements are made, and new features are introduced. Having said that,
  > it’s impossible to ignore the explosion of verbosity that Editions has
  > introduced to the project as a side-effect of this level of available control.

According to Google themselves, this change "solves no user problems" and introduces "an explosion of verbosity."

This blog post wasn’t written by Google themselves, but by Buf, a confusingly-named startup that makes their own protobuf implementation.

I don't understand why this is a problem. It's not uncommon to see multi-decade old systems refactored for extensibility and maintainability every once in a while. What worked when it was initially designed won't work forever, and it's hard to make users lives better if the maintainers can't make their own lives better.

(BTW, this isn't "shipping their org chart.")

I thought Conway's law was fairly prolific.

Though, the closer it is to a true a "law", then the closer GP's comment approaches a tautology.

https://wikipedia.org/wiki/Conway%27s_law

Did it ever work? I remember having to write some pretty gnarly hacks to work around the broken Python code protoc would barf out. Working outside of Go and perhaps Java/Kotlin I've little faith that they're not going to break things further.

As a sibling has already pointed out, this blog post has nothing to do with “Google themselves”. In addition, “nothing really for end users” is obviously bullshit and inconsistent with what the post says itself, e.g.

> This means your existing proto3 projects can now use default field values and extensions.

proto3 not having required and default values was a big source of complaints, so supporting these indefinitely into the future with feature flags instead of having to be stuck forever on proto2 has to be a welcome change for a subset of users, at the very least.

Proto3 already got the required/optional feature added. I don't see why they couldn't have added default values too. (Personally I liked proto3 without optionals, defaults, extensions, etc, but it was a tough sell for people already used to proto2.)

They can, and feature flags is one flexible (if somewhat verbose) way to do it.

I've always liked the simplicity of protobuf, and this change seems to add unnecessary complexity. And: "What problems are Editions solving? In short, nothing really (for end users). The introduction of Editions is the result of major Google-internal refactoring[...]"

Design by promo.

Promobuf

Thinking about this more, this looks overly dismissive. Seems Google wants to be very explicit about future syntax and allow users to mix in different features. I don't think it's worthwhile, but it's doing something.

Is the author just annoyed that Buf now has to put in the work to support editions?

> It’s unclear what an edition's official support lifetime will be today, although Google has suggested it might be “like 10 years.”

ambiguous timeline? sounds right for google.

The timeline depends on when the project will stop being useful to generate promotions.

Thanks Sundar for great org incentive structure!

As if I couldn’t hate protobufs any more. They go and layer on yet more complexity.

These sorts of things need to be as simple as humanly possibly and get out of your way so you can actually do some programming. Instead protobufs are the opposite. Full of backwards compat issues, footguns, and needless complexity that make them an absolute horror to work with.

Do you have a preferred message protocol?

JSON with a good schema if you're doing it across languages?

XML, JSON, and YAML belong to a totally diffent class of use than protobuf, flatbuffers, capnproto, etc

It's like comparing apples and screwdrivers.

They're comparable for a lot of use cases. In fact many protobuf-based APIs will also have a JSON mode.

Exactly because the consumers of those API's exhibit a strong preference for one or the other.

Library/module/API developers often seek to deliver maximimum adaptibility in their interfaces so that consumers of them can satisfy their own more specific/pressing requirements.

imo it's mainly because protos are unusable on the web without something like grpc-web (which requires a proxy) and in general web support in protobufs always seemed like an afterthought, just barely working (if at all) which is kind of ironic considering it came out of Google

I've never used protobufs with web, but I don't see why not. gRPC, on the other hand, requires HTTP/2 which can be a luxury. I remember making a side project that uses gRPC a while ago and trying to deploy on GCP AppEngine, only to find out it doesn't support gRPC.

well if you can't use gRPC which, as you correctly notes, is protocol-incompatible with browser APIs, you kinda need to roll your own RPC framework. Various attempts had been made at this - see connectrpc, for example, by the same company that authored this post. imo, big advantage of protobuf/gRPC ecosystem is that Google is doing all the heavy lifting for you, so unless something else catches on with wider community it's going to be pretty risky beyond hobby projects.

Yeah, that's why I wouldn't do it. Not totally unusable, but painful.

> I've never used protobufs with web, but I don't see why not

We have a bunch of apis like this where I work. It's a huge pain, tbh. You lose the ability to just curl endpoints, or to see what data came over the wire for debugging, not to mention having to use esoteric client libs because every off-the-shelf one just assumes json over http.

JSON is the serialization format. What's the IDL?

I've seen OpenAPI/Swagger in the wild a few times. It's a good way to specify the API for an HTTP server, whether it's JSON or something else. Biggest downside is the confusing name change.

I'm sure there's some great schema implementation that you can bolt onto JSON, but personally I have never seen anyone actually using one in the wild.

With protobuf the schema is essential, it is the foundation. I feel this is a huge advantage especially for communication protocols.

We've used it. It's useful to define the contract between teams.

So you have 2 teams do a meeting, we work out a contract we like, dump that into a jsonschema and you can enforce by unit tests that schema is respected on each side (producer/consumer)

I still don't understand why protobuf has to be in so many Go libraries. It's impossible to avoid unless you roll your own metrics, logging, maybe even mux...

It's simple, lightweight, safe serialization. If you need to convert data structures to bytes, you're going to use protobuf or something comparable.

Even if protobuf isn't your preferred serializer, having any tight binary format be the standard across the ecosystem is miles better than everything slinging JSON around.

Lord knows how much energy we are wasting turning binary data into human readable text that no human will ever look at before it's parsed back into binary.

Performance aside, it's also really nice how a protobuf spec tells you what data structure to expect ahead of time for an API.

The number of APIs with a single JSON example as the only documentation is... unfortunate. Especially if you want to know if a field is nullable.

Taking this a bit further, it's surprising how many companies sell API as a service but provide minimal support for that API, publishing like 1 client/sdk (and often not in the language you need, lol).

Just mindblowing that companies don't adopt a schema-first approach to appeal to a broader customer base. I've lost count of the number of hnd rolled clients I've had to write over the years.

It gets even better if the API itself is backed by some RPC framework. With that, users can simply generate a client and get going.

It feels like we can do better as an industry, this cracked me up because it's so true.

"a single JSON example as the only documentation is... unfortunate"

Funny you should mention nullability. That was a big debate in proto2 vs proto3.

XDR was already a core part of NFS. And then there was a DEC alternative also.

Because the same group in google developed both, indeed.

They were developed as a microservice high-performance lang, compiler, and serialization protocol in the same meetings.

Where are you getting this nonsense from, and if you're just speculating, why are you stating it with such confidence as a fact? Protobufs predate Go by more than half a decade, and there is no overlap between the authors.

Per multiple sources, including the team currently maintaining the Protobuf toolchain within Google, proto3 was largely designed by Rob Pike. Of course the Protobuf wire format is quite a bit older, but some aspects of proto3 and Go's shared semantics (like implicit zero values) do seem to have come from the same mind.

proto1 was designed by Sanjay and Jeff, proto2 by Kenton Varda (who later designed Cap'n'Proto), proto3 I can't remember who but I never heard Rob Pike being credited with this. He did write the very first Go proto binding package though and a separate serialization package called gob with a totally separate rpc library.

That's a much smaller claim, proto3 schemas are a fairly minor evolution over proto2 schemas, mostly removing features. My impression was always that it was either removing things that were expensive to support in Javascript, impossible to serialize to idiomatic JSON, or that the team thought were misfeatures that nobody should ever use. That's a far cry from the original claim that Go and protobufs were co-designed "in the same meetings".

But even that limited claim is kind of hard to believe. Can you link to one of those multiple sources making that claim?

"i came up with the term 'protobuf', and go was not what i had in mind" -rob pike

https://go.dev/blog/gob

You WILL accept The One Right Way to write code and serialize data! -rob pike

Do wake me when they get around to adding the 'high performance' lang part though. Wild how all this ecosystem junk originates from a few early Googlers' fear of learning beautiful C.

Just for posterity, I actually agree with you, and was representing the history of our situation. Not my own opinions. “High performance” absolutely means “buzzword.”

I can attest that the intentions were good, and it’s another tool in our toolbox. A good one, but not the silver bullet sometimes some said.

Because it's useful? What do you propose instead?

Smoke signals

golang is a Google technology, so protobuf is going to be there, right? Moreover golang is compiled and protobuf interoperates best with compiled languages.

Now just wait for golang packages that have conflicting editions requirements!

Google is days away from making Protobuf Editions GA. Never heard of it? Read on.

What’s the meaning behind this comment?

Many folks don't know about what Protobuf Editions are, and it's about to be generally available. I was encouraging people to read the article :)

looks like an LLM spambot

lol, well. Sorry man, I'm a real boy. Check my account age. I can solve some math problems or look at what photos are bridges if you want.

Meh, that's too easy now - if you really want to prove it post something politically incorrect.

Reply on news.ycombinator.com