Hacker News Reader

Comment by zdw | Hacker News Reader

I wonder how this aligns with the protobuf best practice of having the first value be UNSPECIFIED:

https://protobuf.dev/best-practices/dos-donts/#unspecified-e...

When the deserializing half of the protobuf definitions encounter an unknown value, it gets deserialized as the zero value. When that client updates, it will then be able to deserialize the new value appropriately (in this case, "Mint"). The advice on that page also specifies to not make the value semantically meaningful, which I take to mean to never set it to that value explicitly.

> it gets deserialized as the zero value

It’s more complicated:

https://protobuf.dev/programming-guides/enum/

>> What happens when a program parses binary data that contains field 1 with the value 2?

>- Open enums will parse the value 2 and store it directly in the field. Accessor will report the field as being set and will return something that represents 2.

>- Closed enums will parse the value 2 and store it in the message’s unknown field set. Accessors will report the field as being unset and will return the enum’s default value.

Ugh. I hate how we (Google) launched proto editions.

It used to be that we broadly had two sets of semantics (modulo additional customizations): proto2 and proto3. Proto editions was supposed to unify the two versions, but instead now we have the option to mix and match all of the quirks of each of the versions.

And, to make matters worse, you also have language-dependent implementations that don't conform to the spec (in fact, very few implementations are conformant). C++ and Java treat everything imported by a proto2 file as closed; C#, Golang, and JS treat everything as open.

I don't see a path forward for removing these custom deprecated field features, or else we'd have already begun that effort during the initial adoption of editions.

> The advice on that page also specifies to not make the value semantically meaningful, which I take to mean to never set it to that value explicitly.

I've taken to coding my C enums with the first value being "Invalid", indicating it is never intended to be created. If one is encountered, it's a bug.

The example code used added “other” as the last option, which was the source of the problems he described.

This doesn’t happen when you make the first value in the enum unknown/unspecified

No, the problem described in the article is entirely unrelated to where in the enum the Other option is located. There is a different problem where keeping the Other option at the end of the enum changes the value of Other, but that is not the problem that the article is about.

Well it simplifies the logic considerably - if you see an enum value you don’t recognize (mint), you treat it as uninitialized (0).

So any future new flavor will be read back as ‘0’ in older versions.

This is the same as a null pointer, and the requirement is very deeply tied to protobuf as it is used on large distributed systems that always need to handle version mismatch, and this advice doesn't necessarily apply to API design in general.

Even in the simplest web apps you can encounter version mismatch when a client requests a response from a server that just updated.

Hence the advice to make that situation not happen. Update the client and server to support both versions and prefer the new one, then update both to not support the old version. With load balancers and other real-world problems you might have to break that down into 4 coordinated steps.

That only really works if you control the clients, or can force them to update.

> or can force them to update.

I've used a few clients that completely lock me out for every tiniest minor version update. Very top-tier annoying imho.

But it does make the authors' jobs easier.

This implies an API where the server has a single shared implementation. Imagine for instance that the server implements a shim for each version of the interface, then there isn't a need for the null in the API. Imagine another alternative, that the same API never adds a field, but you add a new method which takes the new type. Imagine yet again an API where you are able to version the clients in lockstep. So, it's a decision about how the API is used and evolves that recommends the API encoding or having a null default. However in a different environment or with different practices, you can avoid the null. Of course the reason to avoid the null is so that you can statically enforce this value is provided in new clients, though this also assumes your client language is typed. So in the end, protobuf teaches us, but it's not always the best in every situation.

I don’t mind the zero value for the proto enums, makes sense, but I require converting to my inner logic to not include this “unknown” and error during the conversion if it fails.

I’ve seen engineers bring those unknowns or unspecified through to the business logic and that always made my face flush red with anger.

Why the anger?

If you are consuming data from some other system you have no power over what to require from users. You will have data points with unknown properties.

Say you are tracking sign ups in some other system, and they collect the users’ browser in the process, and you want to see conversion rate per browser. If the browser could not be identified, you prefer it to say ”other” instead of ”unknown”?

I think I prefer the protobuf best practices way: you have a 0 ”unknown”/”unset” value, and you enumerate the rest with a unique name (and number). The enum can be expanded in the future so your code must be prepared for unknown enumerated values tagged with the new (future for your code) number. They are all unique, you just don’t yet know the name of some of the enum values.

You can choose to not consume them until your code is updated with a more recent schema. Or you can reconcile later, annotating with the name of you need it.

Now personally, I would not pick an enum for any set och things that is not closed when you are designing. But I’m starting to think that such sets hardly exist in the real world. Humans redefine everything over time.

I wrote my own Protobuf implementation (well, with some changes). Ditching the default values was one of the changes I made. I don't see any reason to have that. But I don't think that Protobuf is a reasonable or even decent protocol in general. It has a lot of nonsense and bad planning. Having default values is probably not in the ten worst things about Protobuf.

"Unspecified" is semantically different from "other". The former is more like a default value whereas the latter is actually "specified, but not one of these listed options".

Standard practice in protobuf is to never assign semantic meaning to the default value. I think some linters enforce that enum 0 is named "unknown" which is actually more semantically correct than "other" or "unspecified".

Reply on news.ycombinator.com