Go Protobuf: The New Opaque API
go.dev
go.dev
Been increasingly using LSP style JSON-RPC 2.0, sure it's got it's quirks and is far from the most wire/marshaling efficient approach but JSON codecs are ubiquitous and JSON-RPC is trivial to implement. In-fact I recently even wrote a stack allocated, server implementation for microcontrollers in Rust https://github.com/OpenPSG/embedded-jsonrpc.
Varlink (https://varlink.org/) is another interesting approach, there's reasons why they didn't implement the full JSON-RPC spec but their IDL is pretty interesting.
Take JSON-RPC and replace JSON with MsgPack for better handling of integer and float types. MsgPack/CBOR are easy to parse in place directly into stack objects too. It's super fast even on embedded. I've been shipping it for years in embedded projects using a Nim implementation for ESP32s (1) and later made a non-allocating version (2). It's also generally easy to convert MsgPack/CBOR to JSON for debugging, etc.
There's also an IoT focused RPC based on CBOR that's an IETF standard and a time series format (3). The RPC is used a fair bit in some projects.
1: https://github.com/elcritch/nesper/blob/devel/src/nesper/ser... 2: https://github.com/EmbeddedNim/fastrpc 3: https://hal.science/hal-03800577v1/file/Towards_a_Standard_T...
Honestly, if protobuf just serialized to a strictly-specified subset of json, I'd be happy with that. I'm not in it for the fast ser/de, and something human-readable could be good. But when multiple services maintained by different teams are passing messages around, a robust schema language is a MASSIVE help. I haven't used Avro, but I assume it's similarly useful.
Probably the biggest challenge with something like XDR is it's very hard to maintain tooling around it long-term. Nobody wants to pay for forty years of continuous incremental improvement, maintenance, and modernization.
Long term this churn will hopefully slow down, it's inevitable as we collectively develop a solid set of "engineering principles" for the industry.
And yet somehow, we are willing to pay to reinvent the thing 25 times in 40 years.
I'm actually super excited to see how https://www.sovereign.tech turns out long-term. Germany has a lot of issues with missing the boat on tech, but the sovereign tech fund is such a fantastic idea.
- dRPC (by Storj): https://drpc.io (also compatible with gRPC)
- Twirp (by Twitch): https://github.com/twitchtv/twirp (no gRPC compatibility)
Buf distinguishes a few types of plugins: the most important being local and remote. Local plugins are executables installed on your own machine, and Buf places no restrictions on use of those. Remote plugins are hosted on BSR (Buf Schema Registry) servers [1], which are rate limited. All remote plugins are also available as local plugins if you install them.
It's worth to mention that the only time I've personally hit the rate limits of remote plugins is when I misconfigured makefile dependencies to run buf on every change of my code, instead of every change of proto definitions. So, for most development purposes, even remote plugins should be fine.
Additionally, BSR also offers hosting of user proto schemas and plugins, and this is where pricing comes in [2].
[0] https://github.com/bufbuild/buf/blob/main/LICENSE
Me too. My context is that I end up using RPC-ish patterns when doing slightly out-of-the-ordinary web stuff, like websockets, iframe communications, and web workers.
In each of those situations you start with a bidirectional communication channel, but you have to build your own request-response layer if you need that. JSON-RPC is a good place to start, because the spec is basically just "agree to use `id` to match up requests and responses" and very little else of note.
I've been looking around for a "minimum viable IDL" to add to that, and I think my conclusion so far is "just write out a TypeScript file". This works when all my software is web/TypeScript anyway.
State of the art for both gzipped json and protobufs is a few GB/s. Details matter (big strings, arrays, and binary data will push protos to 2x-10x faster in typical cases), but it's not the kind of landslide victory you'd get from a proper binary protocol. There isn't much need to feel like you're missing out.
5-10x is not uncommon, and that's kissing an order of magnitude difference.
That's true of protobufs as much as it is for json, except for skipping over large submessages.
> memory bottleneck
Interestingly, JSON, gzipped JSON, and protobufs are all core-bound parsing operations. The culprit is, mostly, a huge data dependency baked into the spec. You can unlock another multiplicative 10x-30x just with a better binary protocol.
> 5-10x is not uncommon
I think that's in line with what I said. You typically see 2x-10x, sometimes more (arrays of floats, when serialized using the faster of many equivalent protobuf wire encodings, are pathologically better for protos than gzipped JSON), sometimes less. They were aware of and worried about some sort of massive perf impact and choosing to avoid protos anyway for developer ergonomics, so I chimed in with some typical perf numbers. It's better (perf-wise) than writing a backend in Python, but you'll probably still be able to measure the impact in real dollars if you have 100k+ QPS.
Meanwhile if you care about parsing speed, there is MessagePack and CBOR.
If any form of parsing is too expensive for you, you're better off with FlatBuffers and capnproto.
Finally there is the holy grail: Use JIT compilation to generate "serialization" and "deserialization" code at runtime through schema negotiation, whenever you create a long lived connection. Since your protocol is unique for every (origin, destination) architecture+schema tuple, you can in theory write out the data in a way that the target machine can directly interpret as memory after sanity checking the pointers. This could beat JSON, MessagePack, CBOR, FlatBuffers and capnproto in a single "protocol".
And then there is protobuf/grpc, which seems to be in this weird place, where it is not particularly good at anything.
Interestingly, they're not usually smaller than gzipped JSON either (the compression it has built-in is pretty rudimentary), so if you don't compress it and don't have a stellar network you might actually pay more for the total transfer+decode than gzipped JSON, despite usually being somewhat faster to parse.
As far as data dependencies are concerned, you simply can't parse a byte till you've parsed all the preceding bytes at the same level in a message.
A naive implementation would (a) varint decode at an offset, (b) extract the tag type and field index, (c) use that to parse the remaining data for that field, (c1) the exact point in time you recurse for submessages doesn't matter much, but you'll have to eventually, (d) skip forward the length of the field you parsed, (e) if not done then go back to (a).
You can do better, but not much better, because the varints in question are 8 bytes, requiring up to 10 bytes on the wire, meaning AVX2 SIMD shenanigans can only guarantee that you parse 3 varints at a time. That's fine and dandy, except most fields look like 2 varints followed by some binary data, so all you're really saying is that you can only parse one field at a time and still have to skip forward an unpredictable amount after a very short number of bytes/instructions.
If you have more specialized data (e.g., you predict that all field indexes are under 32 and all fields are of type "LENGTH"), then there are some tricks you can do to speed it up a bit further. Doing so adds branches to code which is already very branchy and data-dependent though, so it's pretty easy to accidentally slow down parsing in the process.
Something close to the SOTA for varint decoding (a sub-component of protobuf parsing) is here [1]. It's quite fast (5-10 GB/s), but it relies on several properties that don't actually apply to the protobuf wire format, including that their varints are far too small and they're all consecutively concatenated. The SOTA for protobuf parsing is much slower (except for the sub-portions that are straight memcopies -- giant slices of raw data are fairly efficient in protos and not in JSON).
This isn't the best resource [2], but it's one of many similar examples showing people not finding protos substantially faster in the wild, partly because their protos were bigger than their json objects (and they weren't even gzipping -- the difference there likely comes from the tag+length prefix structure being more expensive than delimiters, combined fixed-width types favoring json when the inputs are small). AFAICT, their json library isn't even simdjson (or similar), which ought to skew against protos even further if you're comparing optimal implementations.
In terms of protos being larger than gzipped json, that's just an expected result for almost all real-world data. Protobuf adds overhead to every field, byte-compresses some integers, doesn't compress anything else, and doesn't bit-compress anything. Even if your devs know not to use varint fields for data you expect to be negative any fraction of the time, know to use packed arrays, ..., the ceiling on the format (from a compression standpoint) is very low unless your data is mostly large binary blobs that you can compress before storing in the protobuf itself.
For a few other random interblags comparisons, see [3], [4]. The first finds protos 3x-6x faster (better for deserializing than serializing) compared to json. The second finds that protos compress better than json, but also that compressed json is much smaller than ordinary protos for documents more than a few hundred bytes (so to achieve the size improvements you do have to "cripple" protos by compressing them).
If you start looking at the comparisons people have done between the two, you'll find results largely consistent with what I've been saying: (1) Protos are 2x-10x faster for normal data, (2) protos are usually larger than gzipped json, (3) protos are sometimes slower than gzipped JSON, (4) when you factor in sub-par networks, the total transfer+decode time can be much worse for protos because of them being larger.
As a fun experiment, try optimizing two different programs. Both operate on 1MB of pseudo-random bytes no greater than 10. Pick any cheap operation (to prevent the compiler from optimizing the iteration away) like a rolling product mod 256, and apply that to the data. For the first program (simulating a simplified version of the protobuf wire format), treat the first byte as a length and the next "length" bytes as data, iterating till you're done. For the second, treat all bytes as data. Using a system's language on any modern CPU, you'll be hard-pressed to get an optimized version of the length-prefixed code even as fast as 10x slower than an un-optimized version of the raw data experiment.
Cap'n proto and flatbuffers (whether gzipped or not), as examples, are usually much faster than both JSON and protobufs -- especially for serialization, and to a lesser extent deserialization -- even when you're parsing the entire message (they shine comparatively even more if you're extracting sub-components of a message). One of them was made by the original inventor/lead-dev of the protobuf team, and he learned from some of his mistakes. "Proper" binary formats (like those, though they're by no means the only options) take into account data dependencies and other features of real hardware and are much closer to being limited by RAM bandwidth instead of CPU cycles.
[0] https://protobuf.dev/programming-guides/encoding/
[1] https://www.bazhenov.me/posts/rust-stream-vbyte-varint-decod...
[2] https://medium.com/@kn2414e/is-protocol-buffers-protobuf-rea...
[3] https://medium.com/streamdal/protobuf-vs-json-for-your-event...
There are just very few spots that actually "need" protobuf at a level of urgency that would justify walking away from self-describing text formats (which is a big, big disadvantage for binary formats!).
[1] Something very well served by JSON
[2] Network routing, stateful packet inspection, on-the-fly transcoding. Stuff that you'd never think to use a "standard format" for.
That means potentially: the majority of devices in the world.
But even there, I find that really these are very big machines in a historical sense. And text parsing is really not that hard, or that big. The first HTTP server was on a 25MHz 68040!
Just use JSON. Anything else in the modern world needs to be presumed to be premature optimization absent a solid analysis with numbers.
There was this one MMO I played, where every packet was just a space separated string, with a fancy variable length number encoding that let them store two digits in a single character.
There is not much difference between
walk <PlayerId> <x> <y>
walk 5 100 300
['walk', '<PlayerId>', '<x>', '<y>']
['walk', 5, 100, 300]
Also JS now has BigInt types and the JSON decoder can be told to use them. So I'd argue it's kind of a moot point at this stage.
The risk with JSON is that too many systems understand it, and intermediate steps can mess up things like numeric precision as well as being inconsistent about handling things out of spec (field order, duplicate fields... etc). This definitely bites people in practice—I saw an experience report on that recently, but can't find the link just now :/
Just one example: Azure Cosmos DB stores JSON documents. If you try to store integers larger than 53 bits there those integers will be silently rounded to 53 bits. I know someone who got burned by this very badly loosing their data; I was able to explain them exactly why...
JSON basically does not define what a number is; and that is a disaster for it as an API format.
I get it, it's probably not a concern for a lot of applications, but when comes to science, games, data it's of big concern... and this excluding the fact that you have to convert back and forth that number a... number of times, and send it on the wire inefficiently - and also miss a way to send it more efficiently using gorilla encoding or something else like that.
JSON is great for a lot of things, but not for high throughput RPC.
I think that you mean the success of JSON APIs. REST is orthogonal to JSON/Protobuf/HTML/XML/S-expressions/whatever.
The parser needs to know when to parse as BigInt vs String.
Binary protocols have exactly same steps.
Redis is also using text based protocol and people don't seem to be bothered too much about it.
The Go OpenAPI did not do this. For many primative types, it was fine. But for protobuf maps, you had to check if the map had been initialized yet in Go code before accessing it. Meaning, with the Opaque API, you can start just adding items to a proto map in Go code without thinking about initialization. (as the Opaque impl will init the map for you).
This is honestly something I wish Go itself would do. Allowing for nil maps in Go is such a footgun.
This was a mistake. You still want to check whether it was initialized most of the time, and when you do the wrong thing it's even more difficult to see the error.
But I do know the footgun of calling "get" on a Java Proto Builder without setting it, as that actually initializes the field to empty, and could call it to be emitted as such.
Such are the tradeoffs. I'd prefer null-safety to accidental field setting (or thinking a field was set, when it really wasn't).
You should only do that if the semantics of the not-set field are different than the default value, which should be rare and documented on the field.
Also everything needs to be marked optional or you'll need to restart the whole service mesh to change the schema.
Agreed that protocol level required is bad, and it's with proto3 made optional the default.
Our team enforces required fields via proto annotations, which can be much more flexible (we have transitional states to be able to upgrade or downgrade the check)
Okay, but look… If I wanted that, I’d create a wrapper library for it and call it a day. But not by default, please.
Like with generics. Now they’re in go. They’re not great, they have some sense but they may as well not exist as far as I’m concerned. I find them pretty useless anyway without lower and upper type bounds.
Given that Go eventually ended up with a very mundane take on generics, in retrospect, it is not at all clear what precluded this same design from being adopted from the get go rather than dragging their feet on it.
Yes, but they weren't good; half of the Java spec and implementations has to deal with its implementation of generics, and the author of Java's generics (Martin Odersky) then moved on to create the even more complicated Scala language. I have this comment [0] and its linked "Java generics FAQ" [1] that is exemplary of its complexity bookmarked for this very occasion.
They ended up with a mundane take on generics because they dragged their feet on it. It's like that quote, "I would have written a shorter letter, but did not have the time." If generics were added without waiting for enough demand, use cases and consideration, it would have ended up weighing them (spec, language, compiler, runtime, etc) down for decades to come. Remember that Go is in it for the long haul.
[0] https://news.ycombinator.com/item?id=9622417
[1] https://angelikalanger.com/GenericsFAQ/FAQSections/TypeParam...
Go ended up with a mundane take on generics because that is consistent with the design philosophy of the language as a whole. My point is that their mundane take on generics was already well-known and well-understood long before Go itself was even a thing, never mind all the rhetoric about "we don't have generics because we don't know how to do them well". If they ended up with some kind of novel arrangement that avoided complexity issues etc present in other generic type system, I would buy that argument. But when they ended up with the design that was there all along, it doesn't really add up.
This is NOT the solution lmao
For most of my projects, I use a web-framework I built on protobuf over the years but slowly got rid of a lot of the protobufy bits (besides the type + method declarations) and just switched to JSON as the wire format. http2, trailing headers, gigantic multi-MB files of getters, setters and embedded binary representations of the schemas, weird import behaviors, no wire error types, etc were too annoying.
Almost every project I've tracked that tries to solve the declarative schema problem seems to slowly die. Its a tough problem an opinionated one (what to do with enums? sum types? defaults? etc). Anyone know of any good ones that are chugging along? OpenAPI is too resty and JSONSchema doesn't seem to care about RPC.
There are lots of other benefits for non performance-oriented teams and projects: the codegen makes it language independent and it's pretty handy that you can share a single data model across all layers of your system.
If you don't care about the wire format, the standard JSON representation makes it pair well with JSON native databases, so you can get strict schema management without the need need for any clunky ORM.
Why aren't they good for that? They can have very high write throughput, they don't require ORMs, and they can be indexed and queried using standard database methods like the SQL language.
You can even enforce strict schemas on them if you want to, just as you would with an RDBMS.
Coral is a schema definition language, yes. But it’s also a full rpc ecosystem.
Smithy at this point is only really an IDL that (in most cases, at least before I left) is “only” used to generate Coral models and then transitively Coral clients and services. The _vast_ majority of Amazon is still on “native” Coral
I'm curious as to why would you think that?
There's a bit of boilerplate in there if you want to use it for a naive implementation, but I don't find it exceedingly resty.
From my POV, using protobuf as a schema declaration tool (as opposed to being a performance tool) is blind follower behaviour. Getting over all the hurdles doesn't seem worth it for the payoff, and it only becomes less valuable when compared to all the OpenAPI tooling you could be enabling instead.
This being for a web-based problem, where we're solving schema declaration.
Huh? From the Swagger site:
"The OpenAPI Specification defines a standard interface to RESTful APIs..."
From their OpenAI Initiative:
"The OpenAPI Specifications provide a formal standard for describing HTTP APIs"
Not sure I care to understand your POV more given this obvious bit was missed by you.
What I'd like is to rewind the time machine and undo all the path-dependent brain damage.
Is there a design doc with the rationale for switching back to explicit presence for Edition 2023?
The closest docs I've found are https://buf.build/blog/protobuf-editions-are-here and https://github.com/protocolbuffers/protobuf/tree/main/docs/d....
This is an important distinction from the proto1->proto2 migration, which was in a much better compatibility situation yet still took years to complete. AFAIK, this is the main reason why the proto team decided to create a superset approach (edition) so migration can be handled as a flag at each message level.
The implicit presence garbage screwed up the API for many languages, not just C++
What is wild is how obviously silly it was at the time, too - no hindsight was needed.
Organizations often promote fools who don’t second guess their beliefs and think they have it all figured out.
From what I can tell, a major source of the problem was that protobuf field semantics were absolutely critical to the scaling of google in the early days (as an inter-server protocol for rapidly evolving things like the search stack), but it's also being used as a data modelling toolkit (as a way of representing data with a high level of fidelity). And those two groups- along with the multiple language developers who don't want to deal with native code- do not see eye to eye, and want to drive the spec in their preferred direction.
(FWIW nowadays I use pydantic for type descriptions and JSON for transport, but I really prefer having an external IDL unrelated to any specific programming language)
Velocity and stability/maturity are in tension, sure. I think for a foundational protocol like protobuf you want the stability and reliability that come from multiple independent implementations more than you want it to be moving fast and breaking things.
> syntax = "proto3" used implicit presence by default (where cases 2 and 3 cannot be distinguished and are both represented by an empty string), but was later extended to allow opting into explicit presence with the optional keyword
> edition = "2023", the successor to both proto2 and proto3, uses explicit presence by default
The root of the problem seems to be go's zero-values. It's like putting makeup on a pig, your get rid of null-panics, but the null-ish values are still everywhere, you just have bad data creeping into every last corner of your code. There is no amount of validation that can fix the lack of decoding errors. And it's not runtime errors instead of compile-time errors, which can be kept in check with unit tests to some degree. It's just bad data and defaulting to carry on no matter what, like PHP back in the day.
How so? In Go, nil is the zero value for a pointer and is ripe for panic just like null. Zero values do not avoid that problem at all, nor do they intend to.
Unfortunately, there is no escaping the extra bits. Not a terribly big deal for a large, powerful machine with lots of memory, but might be a big deal over constrained networks. Presumably that is why proto3 tried to save on the amount of data being transferred. It adds up at Google scale. But it did eventually walk back on that idea, making the extra data opt-in.
struct {
data uintptr
len int
cap int
}
Maps are similar, but have internal data structures that require initialization, thus cannot be reliably used when zeroed. This is probably not so much a performance optimization as convention. You see similar instances in the standard library. For example:
var file os.File
file.Read([]byte{}) // panics; file must be initialized first.Its a performance thing, the map structure requires a heap allocation even with zero elements. This is because the map structure is type generic without being fully monomorphized.
But, that is not the convention. It is well understood that if a struct needs allocations that you must first call a constructor function (e.g. package.NewFoo). It is reasonable to expect that map would stick to the same convention.
However, nil slices work just fine when calling append on them, while writing to a nil map in any way will panic.
For the nil slice though what I said remains true: a nil slice is the same thing as an empty slice. Of course reading from or writing to its first element panics, given that it doesn't have any elements. The same would have happened if you had initialized it as `var ns []int = make([]int, 0)` or `ns := []int{}`.
Ironically, just today I read that in the next release they will add some JSON annotation to distinguish nil slices (omitzero): https://tip.golang.org/doc/go1.24
If what you wanted was to avoid null-panics, you can define the elementary operations on null. Generally null has always been defined as aggressively erroring, but there's nothing stopping a language definition from defining propagation rules like for float NaN.
Is there a way to have your cake and eat it too, and are there real world examples of it?
Pointers are different in that we've decided that the pattern where all bits are 0 is a value that indicates that it's not valid. Note that there's nothing in the definition of the underlying hardware that required this. 0 is an address just like any other, and we could have decided to just have all pointers mean the same thing, but we didn't.
The NULL is just a language construct, and as a language construct it could be defined in any way you want it. You could defined your language such that dereferencing NULL would always return 0. You could decide that doing pointer arithmetic with NULL would yield another NULL. At the point you realize that it's just language semantics and not fundamental computer science, you realize that the definition is arbitrary, and any other definition would do.
As for sum-types. You can't fundamentally encode any more information into an int. It's already completely saturated. What a sumtype does, at a fundamental level, is to bundle your int (which has a default value) with a boolean (which also has a default value) indicating if your int is valid. There's some optimizations you can do with a "sufficiently smart compiler" but like auto vectorization, that's never going to happen.
I guess my point can be boiled down to the dual of the old C++ adage. Resource Allocation is NOT initialization. RAINI.
People generally like to complain about NULL/nil whatever, but they rarely think about what the alternatives mean and what arrangements are completely equivalent. No matter what you do, you have to put some thought into design. Languages can't do the design work for programmers.
In rust, you have:
let s = S{foo: 42, ..Default::default()};
The way you do this is by having types opt in to it, since zero values only make sense in some contexts.
In go, the way to figure out if a type has a meaningful zero value is to read the docs. Every type has a zero value, but a lot of them just nil-pointer-exception or do something completely nonsensical if you try to use them.
In rust, at compiletime you can know if something implements default or not, and so you can know if there's a sensible zero value, and you can construct it.
Go doesn't give you your cake, it gives you doc comments saying "the zero value is safe to use" and "the zero value will cause unspecified behavior, please don't do it", which is clearly not _better_.
Suppose my cake is that I have a struct A which holds a value, that doesn't have a default value, from your library B. Suppose that at the time I want to allocate A I don't yet have the information I need to initialize B, but I also know that I won't need B before I do have that information and can initialize it. In simple terms. I want to allocate A, which requires allocating B, but I don't want to initialize B, yet.
What do I do?
If you answer involves Option<B> then you're asking me to to grow my struct for no gain. That is clearly not _better_.
Go is like PHP in regards to pushing errors forward. You simply cannot validate everything at every step. Decoding with invariants is the right alternative.
it's for different things!
the things I build in Go simply do not need to be robust in the way Rust requires everything to be, and it would be much more effort to use Rust in those problem domains
Is Go a more crude language? maybe! but it lets me GET SHIT DONE and in this case worse really is better.
All I know is that I've spent less time over the last ten years writing Go dealing with NPEs than I have listening to Rust users complaining about them!
if you love Rust so much, YOU use it then! We like Go, in threads about Go. I might like Rust too, in the same way I like my bicycle and my car, if only the cyclists would shut up about how superior their choices are
Option types specifically allow defaulting (to none) even if the wrapped value is not default-able.
You can very much construct null or zero-ish values in such a langage, but it’s not universal, types have to be opted into this capability.
Of course you need some way to represent "absence of a value", the matter is how: simple but incorrect, or complex but correct. And, simple/complex here can mean both the language (so performance tradeoff), and (initial) programmer ergonomics.
That's why I ask if you can have your cake and eat it too, the answer is no. Or you'll get sick sooner than later, in this case.
Except you can. The language runtime is clearly doing it when it stores [None|Some(x)] inline in a fixed size struct.
That's subtlety incorrect. Almost all languages with NULLs in fact already do this, including C. On my machine sizeof(void*)=8, and pointers can in fact express Some(x)|None. The cost of that None is neither a bit not a byte, it's a single value. A singular bit pattern.
See the None that you talk about is grafted on. It wraps the original without interfacing with it. It extends the state by saying "whatever the value this thing has, its invalid". That's super wasteful. Instead of adding a single state, you've exploded the state space exponentially (in the literal sense).
I should also point out that in languages which support both, Option(int*) is a valid construct, and Some(nullptr) is thus not the same thing as None. There could even be valid reasons for needing to do this, such as distinguishing between the JSON objects {} and {"abc": null}. So you can't even use knowledge of built-in types to special-case your generic Option implementation. Even Option(Option(bool)) should be able to represent at least 4 distinct states, not 3: None, Some(None), Some(Some(true)), Some(Some(false)).
It was partially reverted with proto3 optional and fully reverted finally. Go's implementation happened to come around the same time as proto3 so allowed struct access, despite behaving quite differently when accessing nil fields. That is also finally reverted. Hopefully more lessons already learned from the Java days will come sooner than later going forward...
With Go generics - and equivalent in most other languages - you can write a 5-line helper function that takes a string in that format and either returns a valid proto value (using the generic type param to decide which type to unmarshal) or `t.Fatal()`s. You would never do this in production code, but as a way to represent hand-written proto values it's pretty hard to beat.
For several years I used the GoGo Protobuf SDK. It was vastly superior to the awful Javaesque Go code that the official compiler generated. It allowed structs to be pure data structs, was much more performant, and supported a bunch of options to generate native-feeling, ergonomic Go code.
But the Google team refused to partake in any such improvements, and GoGo was shut down as the burden of following the upstream implementation became too big. [1]
I'm not an expert, but as far as I understand, the extra struct junk is mostly to avoid having a parallel set of types for metadata (including reflection). It's unclear to me why these can't simply be generated as internal types with some nice API on top.
Clearly the new field metadata adds to this extra information, and the Go team is moving in the opposite direction of what I thought the future was — they're doubling down on stuffing metadata into the structs, and making the structs bigger and even less wieldy.
I understand how this might make things more performant, but I was hoping this sort of thing could be solved with the type system, especially now that we have generics. For example, surely lazy field access could be done like this:
type Info struct {
User LazyProto[User]
}
userName := user.Get().Name
Then again I've also seen people do these thousand line in-code literal string protos which really grind my gears.
I call this Battlefield versioning, after the Battlefield video game series [1]. I bet the next version will be proto V.
[1]: in order: 1942, 2, 2142, 3, 4, 1, V, 2042
Glad to see this change, for that use case it would've been perfect
Currently I'm not quite sold on RPC since the performance benefits seem to show up on a much larger scale than what I am aiming for, so I'm using a proto schema to define my types and using protoc codegen to generate only JSON marshaling/unmarshaling + types for my golang backed and typescript frontend, with JSON transferred between the two using REST endpoints.
Seems to give me good typesafety along with 0 headache in serializing/deserializing after transport.
One thing I also wanted to do was generate SQL schemas from my proto definitions or SQL migrations but haven't found a tool to do so yet, might end up making one.
Would love to know if any HN folk have ideas/critique regarding this approach.
Now you can not use normal Go struct initialization and you'll have to write reams of Set calls.
type LogEntry_builder struct {
BackendServer *string
RequestSize *uint32
IpAddress *string
// contains filtered or unexported fields
}
func (b0 LogEntry_builder) Build() *LogEntryYou still will need to create temporary objects (performance...) and for an unclear gain.
Underscores are commonly used in names in generated code to avoid conflicts (this applies to all sorts of codegen, not just protobuf). You can easily have both Foo and FooBuilder messages in your protobuf. See also generated enum consts since day one of protobuf-gen-go.
Fair enough but now this name leaks out into code I have to type. I don't dig highly opinionated languages with tooling that complains at me over style guidelines breaking their own style guidelines to solve a problem that they themselves have created.
They painted themselves into a corner and their solution is for me to hit myself in the head with a hammer.
And in this case, perhaps, the resistance and the ugliness of the solution should have made the Protobuf authors to pause for a bit and rethink it.
Several years ago, I used gogoprotobuf instead of regular Go protobuf. It uses normal by-value structures instead of pointers everywhere, with generated code for s11n instead of reflection. It worked about 10x faster that the regular protobuf.
Lots of teams creating rest / json APIs, but very few who use code generation to provide compile-time protection.
Of course, code generation is still practical and I'm a lot more likely to trust a third-party writing a code generator like protobufs, OpenAPI specs, etc, but I would not trust an internal team to do so without a very good reason. I've worked on a few projects that lost hundreds of dev hours trying to maintain their code generator to avoid a tiny bit of copy/paste.
Yet another subtlety is that when cross-compiling, you need to build the code generation tool for the local target always even though the main target could be a foreign architecture. And because the code generation tool and the main code could share dependencies, these dependencies need to be built twice for different targets. That again is something many build tools don't support.
The whole situation is extra confusing because some of the core codegen plugins are built into protoc. That's a distribution and version-matching strategy from the Google team rather than a requirement of the plugin system. I'd very much like Google to ship the built-in plugins as standalone binaries, even if they're also bundled with protoc.
The problem is that each of these standards is different, and impossible to modify.
So, we need something that can serialize one standard to a wire format, and then deserialize from that wire format to another standard.
Oh wait...
A given struct is probably faster for protobuf parsing in the new layout, but the complexity of the code probably increases, and I can see this complexity easily negating these gains.
No, the general idea (and practical experience, at least for projects within Google) is that a codebase migrates completely from one API level to another. Only larger code bases will have to deal with different API levels. Even in such cases, your policy can remain “always use the Open API” unless you are interested in picking up the performance gains of the Opaque API.
message M {
string foo = 1;
}
message N {
M bar = 2;
}
There are some options like nilaway if you want static analysis to prevent you from writing this sort of code, but it's difficult to retrofit into an existing codebase that plays a little too fast and loose with nil values. Having code authors and code reviewers do the work is simpler, though probably less accurate.
The generated code's API has never really bothered me. It is flexible enough to be clever. I especially liked using proto3 for data types and then storing them in a kv store with an API like:
type WithID interface { GetId() []byte }
func Put(tx *Tx, x WithID) error { ... }
func Get(tx *Tx, id []byte) (WithId, error) { ... }
It’s good to isolate your dependencies within the code :)
However I can get behind it for the lazy decoding which seems nice, though I doubt its actual usefulness for serious software (tm). As someone else already mentioned, an actual serious api (tm) will have business-scope types to uncouple the api definition from the implementation. And that’s how you keep sane as soon as you have to support multiple versions of the api.
Also, a lot of the benefits mentioned for footgun reductions smell like workarounds for the language shortcomings. Memory address comparisons, accidental pointer sharing and mutability, enums, optional handling, etc are already solved problems and where something like rust shines. (Disclaimer: I run several grpc apis written in rust in prod)
To bake endianess and alignment requirements into your protocol.
This one small change can result in an order of magnitude improvement.
- Messages (especially opaque ones) are not supposed to be copied. The
recommendation is to use `m := &mypb.Message{}`.
- This would make migrating to use the opaque API more difficult, if the
getters don't return the same type as the old open API fields, much more
code needs to be rewritten, or some wrapper that allocates a new slice on
every get.
- Users expect that `subm := m.GetSubMessages()[2] ;
m.SetSubMessages(append(m.GetSubMessages(), anotherm))) ; subm.SetInt(42) ;
assert(subm.GetInt() == m.GetSubMessages()[2].GetInt())`. This would not be
the case if the API returned a slice of values.
- ...
If we had returned an iterator (new in Go 1.23) instead of an actual slice, then it would've been possible to vary the underlying representation (slice-of-pointers, slice-of-value-chunks, ...). But there are other downsides to that too:
- Allocations when passing iterators to functions that expect a slice.
- Extra API surface for modifying the list (append, getn, len, ...).
Another thing that could be considered is: when decoding, allocate a slice of values ([]mypb.Message), *and* a slice with pointers (or do it lazily): []*mypb.Message. Then initialize:
for i := range valuel {
ptrl[i] = &valuel[i] // TODO: verify that this escape doesn't cause disjoint allocations.
}
So?
> - This would make migrating to use the opaque API more difficult
The opaque API is stupid to begin with. Now the objects are no longer threadsafe. You can't just read a message in one thread and process it in two different threads.
> Users expect
Then don't expect this. If you're breaking the API, then at least break it in a way that makes it better afterwards.
This is not correct. The Opaque API provides the same guarantees as before, meaning you can read a message in one goroutine and then access it (but not modify it) from other goroutines concurrently.
> So?
If you have a []mypb.Message, and range over it in the normal way:
if _, m := range msgs {
// Use.
}
Why?
An example w.r.t. the surprising semantics:
var ms []mypb.Message = ppb.GetMessages() // A repeated submessage field
for i, m := range ms {
m.SetMyInt(i)
}
assert(ppb.GetMessages()[1].GetMyInt(1) == 1) // This would fail in general, due to SetMyInt acting on a copy.
You may have noticed that changing a heavily pointerized tree of types into value types often compiles with just a few changes, because Go automatically dereferences when needed. But it often won't work from a semantic point of view because the most intuitive way to modify such types uses copies (the range loop is a good example).
Now imagine changing the representation such that it carries a mutex, or another nocopy type. That would lead to issues unless those nocopy types would be encapsulated in a pointer. But then you get issues with initialization:
var m mypb.Message // Value type, but what about the *sync.Mutex contained deep within?
func process(lm mypb.LazyMessage) {
if lm.GetSubMessage().GetInt() != 42 {
panic("user is not enlightened")
}
}
var lm mypb.LazyMessage
process(lm) // Copy a fairly large struct.
ln.GetSubMessage().GetInt() // Does lazy unmarshaling of sub_message redundantly.
ms := m.GetSubMessages() // []mypb.Message
el := &ms[0]
anotherEl := new(mypb.Message)
ms.SetSubMessages(append(ms.GetSubMessages(), anotherEl)) // Can cause reallocation, now el no longer references to m.GetSubMessages()[0]. But it no reallocation happened, it does.
Since you seem so sure of your position, I'm actually curious. How would you design the API, how would you use it? Do you have any examples I can look at of this style being used in practice?
I certainly won’t allow this to be used by the engineering teams under me.
They are called `GetFoo()` instead of the idiomatic `Foo()`, but that is to ensure compatibility with the API where the fields are directly exposed as `Foo`, which also makes sense.
You can build services internally with gRPC and serve a public graphQL API that aggregates them.