The problem went away with all optional fields so it was decided the headache wasn't worth it.
I suspect that not having nullable fields, though, is just a case of letting an implementation detail, keeping the message representation compatible with C structs in the core implementation, bleed into the high-level interface. That design decision is just dripping with "C++ programmers getting twitchy about performance concerns" vibes.
Then proto3 went and implemented a hybrid that was the worst of both worlds. They made all fields optional, but eliminated the introspection that let a receiver know if a field had been populated by the sender. Instead they silently populated missing fields with a hardcoded default that could be a perfectly meaningful value for that field. This effectively made all fields required for the sender, but without the framework support to catch when fields were accidentally not populated. And it made it necessary to add custom signaling between the sender and receiver to indicate message versions or other mechanisms so the receiver could determine which fields the sender was expected to have actually populated.
This is very solid with message types, but for basic types you can add `optional field` if needed as well (essentially making the value nullable)
Proto3 made primitives all non-optional, default 0. But messages were all still optional, so you could always wrap primitives that really needed to be optional. Then they added optionals for primitives too recently, implemented internally as wrapper messages, which imo was long overdue.
Protobufs generally respect this property, but required fields break it. If you have a field that is marked 'required' and then a message omitting it passes through a server with that schema, the whole message will fail to parse. Even if the schema definition on both the sender and the recipient omits the field entirely.
Consider what happens when you add a new required field to a protobuf message, which might be embedded deep in a hierarchy, and then send it through a network of heterogenous binaries. You make the change in your source repository and expect that everything in your distributed system will instantly reflect that reality. However, binaries get pushed at different times. The sender may not have picked up the new schema, and so doesn't know to populate it. The recipient may not have picked up the new schema, and so doesn't expect to read it. Some message-bus middleware did pick up the new schema, and so the containing message which embeds your new field fails to parse, and the middleware binary crashes with an assertion failure, bringing down a lot of unrelated services too.
You could (e.g.) annotate all key fields as IDENTIFIERs. Client code can assume those will always be set in server responses, but are optional when making an RPC request to create that resource.
(This may just work in theory, though – I’m not sure which code generators have good support for field_behavior.)
I think the main problem with it, is that you cannot distinguish if the field has the default value or just wasn't set (which is just error prone).
However, there are solutions to this, that add very little overhead to the code and to message size (see e.g. [1]).
I don't get what part of your link I'm supposed to be looking at as a solution to that issue? I wasn't aware of a good solution except to have careful application logic looking for sentinel values? (which is garbage)
Re: your question, for proto3 an field that's declared as "optional" will allow distinguishing between set to default vs. not set, while non-"optional" fields don't.
Required/validation is for application.
We dealt with a bug that for weeks was silently corrupting the results of trials pitting the performance of various algos against each other. Because a valid response was "no reply/opt out", combined with a bug in processing the "opt out" enum, also combined with a bug in score aggregation, functions were treated like they replied "0.0" instead of "confidence = None".
It really should have defaulted NaN for missing floats.
Wait, are you using Python?
double lat = 1; // Geodetic latitude, decimal degrees
double lon = 2; // Geodetic longitude, decimal degrees
"Hmm, lat = 0, I guess they didn't fill out the message. I'll thrown an exception and handle it as an api error"
[Later, somewhere near the equator]
"?!?!??!"
------------------
"Ok, we learned our lesson from what happened to our customers in Sao Tome and Principe: 0.0 is a perfectly valid latitude to have. No more testing for 0, we'll just trust the value we parse.
[Later, in Norway, when a bug causes a client to skip filling out the LatLon message]
"Why is it flying to Africa?!?!"
------------------
Ok, after the backlash from our equatorial customers and the disaster in Norway, we've learned our lesson. We will now use a new message that lets us handle 0's, but checks that they really meant it:
message LatLonEnforced {
optional double lat = 1;
optional double lon = 2;
[At some third party dev's desk] "Oh, latitude is optional - I'll just supply longitude"
[...]
"It's throwing exceptions? But your schema says it's optional!"
------------------
Ok, it took some blood sweat and tears but we finally got this message definition licked:
message LatLonEnforced {
optional double lat = 1; // REQUIRED. Geodetic latitude, decimal degrees
optional double lon = 2; // REQUIRED. Geodetic longitude, decimal degrees
[Later, in an MR from a new hire] "Correct LatLon doc strings to reflect optional semantics"
Edit: If a client doesn't fill out the LatLng message, that's different from lat and/or lon being null or 0. The whole LatLng message itself will be null. Proto3 always supported that too. But it's usually overkill to check the message being null, unless you added it later and need to specially deal with clients predating that change. If the client just has a bug preventing it from filling out LatLng, that's the client's problem.
The confusing part here is that even if the LatLng message is null, LatLng.lat will return 0 in some proto libraries, depending on the language. You have to specifically check if the message is null if you care. But there are enough cases where you have tons of nested protos and the 0-default behavior is actually way more convenient.
Def +1 the confusingness of how you have to explicitly check for has_latlon() to make sure you're not just getting default values of a non-existent LatLon message. The asymmetry between primitive and message-type fields in having explicit presence checking is also part of my beef. It's weird to have to teach junior devs that you can do has_* checks, but only for messages.
I can imagine message nullness being clearer in languages with optional unwrapping like JS. Like foo.bar.baz gives an error if bar is null, and if you want default-0 you use foo.bar?.baz. Idk if that's what happens though.
If you really feel like expressing that LatLon as possibly null, it should rather be:
message User {
optional LatLon position = 1;
} astring = sql.nullstring{isNull=true, value=""}
if astring.isNull {don't do anything}
else {process astring.value}
if packet.HasField(astring) {then process packet.astring}
Correct answers: 1.0, 0.0, 1.0
Confidence from algo: 1.0, 0.0, n/a
Confidence on the wire: 1.0, 0.0, 0.0
Score after bug: 66%
Score as it ought to be scored: 100%
It was enough to make several algorithms which were very selective in the data they would attempt to analyze (think jpg vs png images) went from "kinda crap" in the rankings to "really good"
If you had use 3 float values in Go or Java you would have had the same problem.
Tangent: I've seen an antipattern of using enums to describe the "type" of some polymorphic object instead of just using a oneof with type-specific fields. Which gets even worse if you later decide something fits multiple types.
I have a hard time thinking of places I really need both 0 and none. The only example that comes to mind is building room numbers, in some room search message where we wanted null to mean wildcard. In those cases, it's not hard to wrap it in a message. Probably the best argument for optionals is when you have a lot of boolean request options where null means default, but even then I prefer instead naming them such that the defaults are all false, since that's clearer anyway.
It did take some getting used to and caused some downstream changes to how we design APIs. I think we're better for it because the whole null vs 0 thing can be tedious and error-prone, but it's very opinionated.
Here's another fun one: I've seen APIs where "0.0" was treated as "no value, so take the default value". The default value happened to be 0.2.