When nil is not nil | Hacker News Reader

When nil is not nil

bitmaybewise.substack.com

45 pointsbitmaybewise68 comments

"My favorite is always the billion dollar mistake of having null in the language. And since JavaScript has both null and undefined, it's the two billion dollar mistake." -Anders Hejlsberg

"It is by far the most problematic part of language design. And it's a single value that -- ha ha ha ha -- that if only that wasn't there, imagine all the problems we wouldn't have, right? If type systems were designed that way. And some type systems are, and some type systems are getting there, but boy, trying to retrofit that on top of a type system that has null in the first place is quite an undertaking." -Anders Hejlsberg

https://news.ycombinator.com/item?id=19568681

https://news.ycombinator.com/item?id=19568378

https://www.youtube.com/watch?v=csL8DLXGNlU

> "And some type systems are, and some type systems are getting there, but boy, trying to retrofit that on top of a type system that has null in the first place is quite an undertaking." -Anders Hejlsberg

That's one thing I will heartily credit C# for, despite not being a great fan of the language. Actually retrofitting null-safety in a language never designed for that was an impressive undertaking.

Anders Hejlsberg was talking about TypeScript in that $2 billion quote, which he also designed.

I love programming in C# myself, and think it's quite a nice language, carefully designed to say "Fuck You!" to Java, the same way Java was carefully designed to say "Fuck You!" to C++, which itself was insanely designed to say "Fuck You!" to C.

Whereas TypeScript, which I also enjoy and appreciate, was carefully designed to say "Fuck You!" to JavaScript, but JavaScript was less designed than just haphazardly thrown together as LiveScript, and then deceptively renamed and marketed to say "Me Too!" to Java.

Honestly, nil is not so bad in Elixir. (It's an atom type, and it and false are the only falsely values, and there are some nice defaults like List.wrap(nil) == [])

Found out the hard way with Javascript I have to be careful about just doing "if(blah)" to represent what I thought was shorthand for "if(blah !== undefined && blah !== null)", when I had a weird bug where the logic was being skipped, and discovered that, oh, that also means "if(blah !== 0)"! The value was sometimes 0 and that was also causing it to evaluate to false, when I expected that to be true.

Now to be safe I'm always explicit when checking for undefined and null values, and also try to limit how I use them.

> that also means "if(blah !== 0)"!

The empty string ("") is also falsy. So is `false`, obviously. And the bigint 0. And NaN. Here is the complete truth table: https://262.ecma-international.org/#sec-toboolean

FWIW undefined and null are loosely equal to one another, it's one of the few cases where loose equality is legitimate. As in

    if (blah != null)

will exclude null and undefined, and only those two[0]. That's step 2 and 3 of IsLooselyEqual: https://262.ecma-international.org/#sec-islooselyequal

[0] and Document.all but if you're dealing with that you're beyond the reach of god

https://www.reddit.com/r/ProgrammerHumor/comments/225i15/pro...

"Proof that Brendan Eich never really cared about equality."

https://i.imgur.com/j7u6KbR.png

If we're doing thunderdome, JS doesn't beat C++: https://howardhinnant.github.io/smf.jpg

By the time you're declaring copy and move constructors you're much deeper than equality checks.

My answer to this recurring question: http://www.jerf.org/iri/post/2957

The really short answer is that the problem stems from a incorrectly imported concept from C, where void pointers are all 0. If you treat Go as Go, and don't try to understand it as another language, you would understand there's no particular reason to treat nil the way people are treating it. It isn't a void pointer. There's also an issue where people incorrectly attribute the problem; it isn't where the nil (of one type) fails to compare equal to nil (of another type), it's where the value was created in the first place that didn't mean what a C-based understanding thought it meant. Go isn't C, just like any language X isn't language Y.

In reality, once you understand this issue correctly, it essentially dissolves simply because you have the correct understanding. I've been programming in Go now for years and I simply don't hit this problem. You do have to be willing to let go of your wrong ideas, but, it's not a matter of opinion about design or something; they are objectively wrong ideas about how the values in Go work. Once you move the problem from trying to "detect" the problem to not creating the problem in the first place, it's not hard.

The real problem is that they imported the concept of null/nil being a value of every type from C. Null/nil is fine (although Option/Optional/Maybe is better when you want to nest), but it should be an opt-in part of the type such that it is a compile time error to assign a null/nil value to a variable unless it's the nullable variant of the type.

Actually, not all Go types are nilable. Integers and strings; those things aren’t nilable. Pointers, interfaces, slices, and some other stuff is. I’m new to Go so I don’t remember the rules exactly, but you can look up the exact details I’m sure.

> Integers and strings; those things aren’t nilable.

Which is then annoying because if you want an optional integer you now need a pointer, and a heap allocation.

does anyone else think Go would be a much better language without the pointer distinction? maybe an explicit mutable type modifier instead?

Yea, I think references to other values should be a type, like a `Reference[T]`. Then you can pass something by reference, rather than value, without having confusing referencing/dereferencing. A `Reference[T]` could have some sort of element reflection, where you can just use it like it's a `T`.

Pointers, referencing, dereferencing; that can get confusing even for skilled engineers.

That's one of the most annoying things to me coming from Python: the inability to have a default argument value.

Go can use escape analysis to determine if a heap allocation is actually required in some cases.

Pointers are conflated with Option-types in go. Understandably, because go wants to be as simple as possible, has those nasty zero-values as "solution", had no generics for a long time, and will probably never have sum-types.

So, guess if it's a pointer or an Option, and hope that your peers guess the same.

"The real problem is that they imported the concept of null/nil being a value of every type from C."

While I agree that's a problem, it's mostly a separate problem to the nil interface vs. interface containing a nil. The "nilness" provides a cognitively-convenient hook, especially with the attractive nuisance of the whole issue of nil/null in general, but the root cause of this particular issue is really "putting things into interfaces that don't fulfill the contract of the interface", which as I show in my post, doesn't require a nil anywhere in sight to do it.

When you stop doing that, this whole interface nil vs. containing a nil problem goes away. Basically, in Go code, you should never be trying to penetrate an interface value to see if it contains a nil. If you're doing that, the error occurred when the value you are examining was created, not at the point of use.

In fact, stop putting values that can't implement interface contracts into interface values and other code issues just go away, too. Not just in Go, either. Most large dynamically-typed language programs I've ever worked in are also shot through with this exact issue, it's just less visible because the compiler doesn't help you with the interfaces. They're still there, though. Just in the dark.

Even as a beginner in Go, I somehow internalized that if you want a value to be `nil`, use a "bare `nil`" rather than a concrete value initialized to `nil`. I don't know why you would want to initialize something on the heap, like in the article, if you didn't plan to use it later. Maybe in the most extreme case you might want to prepare an error object before the processing occurs, but even then I have never seen any Go tutorial that didn't return a "bare `nil`" for the `(err error)` return value if there was no error being returned. I guess people coming to Go from other languages sometimes don't look at the tutorials, or perhaps don't try and understand the language's "best practices"?

In real code, the problem would usually occur not because you want a value to be nil but because you’re converting a value you already have to an interface. And that value happened to be a pointer which is nil.

Arguably the most common line of code in Go, ‘if err != nil’ is doing what you’re saying you should never do in the language: check if an interface value is nil. (And, in fact, care must be taken when returning custom error types precisely because of the typed nil issue.)

So I don’t think the authors of the language see it the same way you do.

(FWIW I was also a Go style reviewer working with the Go team for a few years, and I’ve never heard this idea from them in that time.)

So can you define what it is, beyond "not like a C void pointer"?

Because effectively, basic pointer behavior (nil check) is dangerously (panics) different depending on returning a nil interface or nil *struct (both are interchangeable).

Granted, I don't run into this problem every day, but it's not due to "understanding the issue correctly".

> C, where void pointers are all 0

I think you mean "null pointers are all 0". void pointers are something entirely different, and certainly not always 0.

> In reality, once you understand this issue correctly, it essentially dissolves simply because you have the correct understanding.

When you understand something then you don’t have a problem understanding it any more.

He's saying theres a simple mental model where this all makes sense. That's not always the case, despite your implication.

Reading the original FAQ: Am I wrong or is this (yet another) abstraction leak even more confusing than C? How much longer does everyone cling to the "What I like about Go is that it's so simple!" mythology? Bah, here comes the downvotes...

Well I’m here to agree with both sides and not downvote you. :). I think the nil issue is one of the ugliest, most confusing parts of Go, and yet I still like the language as a whole for its relative simplicity.

I think this one difficult subject does not take away from the common feeling that overall Go is simple. This is like looking at a small scratch on the car and claiming the entire car is damaged.

“Simple” means the definition has been kept compact, which should suggest problems went unaddressed. Gabriel’s “Worse is Better” essay reminds us that making a system do the Right Thing™ is rarely simpler.

But making the runtime go out of its way to store the exact type of an object you don’t have is damn weird. In theory a method could accept a nil receiver, but in practice they never seem to.

This reminds me of a footgun in LuaJIT's FFI: nil is generally a falsy value in Lua; a null pointer returned from a C API is equal to nil, but is not a falsy value in LuaJIT.

This means that

    if x and x == nil

evaluates to true - a highly unusual condition in Lua. It makes sense if you think about it for a bit, but it really makes you scratch your head when you encounter this situation for the first time.

From what I've found maintaining https://github.com/Planimeter/game-engine-2d is that you should be explicitly checking for nil, separately from checking for false-evaluating values.

We found that this had the added benefit of semantically checking for something that did not exist, versus some contextual "false" value.

I suspect this is a good practice in other languages as well.

When does

    if x and x == nil

ever evaluate to true in LuaJIT? What does the FFI have to do with it? Please explain.

Edit:

I think I see what you mean here. The x variable is actually a CDATA object returned by the LuaJIT FFI. Upon comparison, the NULL pointer value inside would be converted to nil. I don't see this as being a 'footgun' unless you forget that you're dealing with a CDATA object.

A null pointer is a boxed value in LuaJIT, therefore not nil. But it is equal to nil when compared with nil.

    local x = require('ffi').cast('void*', nil)
    if x and x == nil then
        print("x is truthy and nil at the same time")
    end

Unity3D also introduces null-like C# objects that aren't really null, for wrapping C++ objects that can disappear behind C#'s back. This is related to Unity's own C++ object wrappers derived from UnityEngine.Object, not C#'s standard P/Invoke FFI.

So of course standard C# can report a NullReferenceException if you try to do something with an actual "null" value, but in Unity3D you can also have object references to C++ object wrappers (like GameObject) that override == and != to act like they're null in some cases, but report a different MissingReferenceException if you try to do other things with them. And that makes comparing Unity C++ objects to null much slower, since it has to thunk into native code to perform the comparison, not just use one machine instruction.

This causes a some very subtle bugs (and performance problems) because many people don't actually understand what's really going on behind the scenes. They don't realize there's another level of indirection and overriding going on, and just brute-force a solution by checking for null more often, without realizing the people getting confused really do have a valid reason for their confusion and aren't just sloppy programmers.

Notice how some of the forum replies in the links below just mansplain to the people reporting problems that they "simply" need to check for null harder as if they were dummies, without mentioning Unity's obscure == and != C++ object wrapper overriding hack that's the root of the problem, even though they did check for null (i.e. before calling a coroutine, which was too early), but it unexpectedly changed out from under them. This is a classic leaky abstraction and foot gun.

https://blog.theknightsofunity.com/story-nullreferenceexcept...

>But Unity objects are a bit different. Unity objects can report themselves as null even if they are still referencing an object!

>No, that’s not a bug! Some referenced objects are scene objects. Imagine what happens if a scene is being replaced by another scene – all regular scene objects have to be destroyed. Since Unity cannot update your valid references inside your code (they still will be valid after destroying the scene), it is faking a null check for those that you are still referring to, so it will (should) stop you from use these any longer. In case you do try to use any of these objects, you may receive a message like this one:

>MissingReferenceException: The object of type 'Texture2D' has been destroyed but you are still trying to access it. Your script should either check if it is null or you should not destroy the object.

So you can have a non-null reference in a variable, that apparently changes to a (virtual) null behind your back, without you ever changing the value of the variable! Here's an example of somebody who was confused by that, because they checked for null BEFORE entering a co-routine, and then even though they never reassigned the value of the variable in the co-routine, the GameObject that it referred to got destroyed, morphing the C++ GameObject wrapper into a virtual null reference behind their back. (i.e. an actual C# wrapper object that lies that it's equal to null, because the underlying C++ object it points to has been destroyed.)

It gets even more confusing when you mix it with coroutines and C# native implicit null checking operators ?. and ??.

https://forum.unity.com/threads/missingreferenceexception.17...

>Fake-null objects are an important requirement due to the fact that Unity is a C++ engine with explicit memory management and C# / .NET is a managed environment. Objects derived from UnityEngine.Object have an actual partner on the native code side. You can not destroy objects explicitly in the managed world. However we have the Destroy method to destroy any objects derived from UnityEngine.Object. What the method does is actually destroying the object on the native side and mark the managed wrapper object as "being destroyed". Such an object will essentially pretend that it is null since it's no longer useable since the native counterpart is missing. That fake null object will eventually be garbage collected once all references to it are gone.

Here's a good explanation from Lucas Meijer at Unity:

https://blog.unity.com/technology/custom-operator-should-we-...

>Custom == operator, should we keep it, by Lucas Meijer.

>[...] Going over all these upsides and downsides, if we were building our API from scratch, we would have chosen not to do a custom null check, but instead have a myObject.destroyed property you can use to check if the object is dead or not, and just live with the fact that we can no longer give better error messages in case you do invoke a function on a field that is null.

>[...] We're a bit nervous about that, as if you haven't read this blogpost, and most likely if you have, it's very easy to not realise this changed behaviour, especially since most people do not realise that this custom null check exists at all.[3] [...] This led to the "well if even we missed it, how many of our users will miss it?", which results in this blogpost :)

It's a common problem raised on the forums:

https://answers.unity.com/topics/missingreferenceexception.h...

https://answers.unity.com/questions/1705335/how-can-throw-mi...

https://stackoverflow.com/questions/58607172/missingreferenc...

https://forum.unity.com/threads/missingreferenceexception-ar...

https://forum.unity.com/threads/missingreferenceexception-i-...

Also note that null comparisons against Unity objects are considered "expensive" because of this hidden overriding of the equality operator.

https://blog.jetbrains.com/dotnet/2019/02/21/performance-ind...

https://github.com/JetBrains/resharper-unity/wiki/Avoid-null...

>Avoid null comparisons against UnityEngine.Object subclasses

>Classes deriving from Unity.Object inherit equality operators that change the behaviour of the == and != operators. While these operators will perform standard .NET reference equality, if comparing one side to null, these operators will call native code to check if the underlying native engine object is still alive. For more details on the background of this, please see the explanation of the "Possible unintended bypass of lifetime check of underlying Unity engine object" inspection.

>This transition to native code can be expensive, as Unity will perform lookups and validation to convert the script reference to the native reference. While small, the cost of comparing a Unity object to null is much more expensive than a comparison with a plain C# class, and should be avoided inside a performance critical context, and inside loops.

>This inspection will add a performance indicator highlight to null comparisons against UnityEngine.Object subclasses inside a performance critical context. It will also provide the following Alt+Enter context actions:

>Move outside loop. This will introduce a variable to hold the result and move the comparison outside the scope of a loop. Move to Start or Awake. This will introduce a field to hold the result and move the comparison to Start or Awake. This inspection was first added in Rider/ReSharper 2018.3

https://github.com/JetBrains/resharper-unity/wiki/Possible-u...

>Possible unintended bypass of lifetime check of underlying Unity engine object

>This is a Unity specific inspection. It only runs in a Unity project.

>This warning is shown if a type deriving from UnityEngine.Object uses either the null coalescing (??) or null propagation or conditional (?.) operators. These operators do not use the custom equality operators declared on UnityEngine.Object, and so bypass a check to see if the underlying native Unity engine object has been destroyed. An explicit null or boolean comparison, or a call to System.Object.ReferenceEquals() is preferred in order to clarify intent.

Empirically, having an “unset” value is pretty important. It’s not nil or null or undefined, but rather unbound. A sentinel value.

It pops up everywhere in python libraries. Core libs, like inspect.py.

I’ve never seen a good treatment of unset. There are a bunch of interesting corner cases, like how to store an unset value, or whether setting a hash map entry to unset should delete the entry, or whether it’s falsey. (Falsy?)

You want it so that you can tell whether an argument to a function wasn’t passed at all, vs whether someone explicitly set it to nil.

> It pops up everywhere in python libraries.

It pops up everywhere because Python has default values and sentinel objects.

In a different language with a different logic, it wouldn't pop up anywhere as much. And usually (though not always) it doesn't overlap with the need for a null (and / or the distinction doesn't matter).

When it does, you can just add a layer of indirection e.g. `Maybe (Maybe a)`. That's what you get if you store an option in a map, in a language with option types:

    lookup :: Ord k => k -> Map k a -> Maybe a

so if you have a

    Map k (Maybe a)

(a map from any key to an optional value, aka your standard Java or JS or Python map) lookup will return a `Maybe (Maybe a)`, the outer Maybe is the result of the lookup, and the inner Maybe is the stored value. There is no information loss, and if you don't care about the distinction you can just `join` the result in order to collapse it.

> Empirically, having an “unset” value is pretty important.

Being able to check if some identifier isn’t bound is useful, but as soon aa you do so by means of an “unset value”, you need abother way to detect if it was really unset or just set to the unset value.

> It pops up everywhere in python libraries

It pops up everywhere in Python libraries as a means of hacking around the fact that function defaults are evaluated at compile time, so, particularly with mutable defaults, you end up with a shared object rather than a fresh one with each default invocation, so a sentinel value is used to know you need to create a fresh value in the function body rather than using an default thar gets improperly shared.

Arguably, what Python mostly needs to streamline this isn’t a systematic handling of sentinel values but syntax for specifying that the default value is the return value of a specified callable rather than a specific function-definition-time value.

> Being able to check if some identifier isn’t bound is useful, but as soon aa you do so by means of an “unset value”, you need abother way to detect if it was really unset or just set to the unset value.

"Unset" has that problem. But a sentinel value for "unbound" is very doable under certain limitations. A language can have "being unbound" be exclusive to data structures, and "containing the sentinel value for being unbound" be exclusive to local values. Then it's never ambiguous. This also implies you won't have a "locals()" equivalent.

The concept of "there is no value here" is very useful. For example when trying to get a value from a hash map, or having optional parameters.

But nil/null/undefined doesn't seem like the best way to go about this when designing a language. Not every value is optional, but if anything can be nil then you will spend a lot of unit tests and mental overhead on managing that disconnect. Systems that allow you to explicitly specify what can have no value (like SQLs NotNull, or Rust's or c++'s Optional) seem to do much better in terms of preventing bugs.

The problem with "unset" being a value is that it can then get propagated very far (so long as it just gets copied and not used) before blowing up on access. And then you're looking at the value and have no clue as to what logic error produced it in the first place.

This is easily accomplished in python:

    _DEFAULT = object() # bonus points: make this reload safe

    def frobnicate(bar=_DEFAULT):
        if bar is _DEFAULT:
            ...

But really the main reason this comes up is because python went with the dumb decision to evaluate default args at defintion and not call time (unlike, say, common lisp etc.).

But in this instance, Python's dumb evaluation evaluation strategy for default expressions makes your code unnecessary, because you could just have:

   def frobnicate(bar = object()):    # object() evaluated once, at definition time

Python behaves as if by a transformation to:

   __hidden_global_42 = object();  # evaluation at definition time

   def frobnicate(bar = __hidden_global_42):

which is exactly what you have done manually.

But then how does the body of the frobnicate function tell if bar is the default sentinel object() value or not (if bar is ???), without somehow using reflection to fish out a reference to the default value of bar from the function definition? Or is there an easy and efficient way to do that? Add yet another optional-but-not-meant-to-be-passed parameter bar_default=bar maybe? Yuck.

Looks like you can get at that value in frobnicate.__defaults__.

Having spread that, I have to go wash my hands now.

https://en.wikipedia.org/wiki/Falsies

The easiest way to remember this for your own use is to remember that, in Go, the == equality operator is a bitwise comparison. It will be true if and only if two entities are bitwise equal.

In Go, every value is stored as a tuple: (concrete type, value). So (int32, 5) is not the same as (int64, 5), even though the numeric value is the same.

In general, the compiler will prevent comparisons that are always going to be false, such as comparing an int32 to an int64: https://go.dev/play/p/q0skdfUnWsD

The only time when you'll be able to compare two entities with identical values but different concrete types (and have your code compile) is when they're being compared through variables stored in an interface: https://go.dev/play/p/t0QDFh70Ut4.

This can be surprising when you first encounter it, but once you understand how it works, it's easier to remember.

> the == equality operator is a bitwise comparison

With floats being the exception

And strings being the second exception.

And complex as they're basically a pair of floats.

And transitively any array or struct containing one of the exceptions.

So really, == is mostly not a bitwise comparison, when you think about it.

What do strings do with == that isn't bitwise over the contents?

Compare the contents of the actual buffer.

If == on string was a bitwise comparison, it would be an identity check. But two different strings with the same content compare equal, meaning go dereferences the buffer pointer.

So it is comparing the contents exactly, the same as with an array or struct? In that case strings aren't an exception, looking at contents is just how structs work.

Oh hey, flying goalposts, cool.

The hell are you talking about? I used the same phrasing both times.

You're the one who brought up comparing the contents instead of just the types and pointers. The first pair of goalposts was the fixed size comparison of the (type, pointer) bits of the string object reference, the second pair of goalposts was the variable length and arbitrarily long contents including the bits that the two different string pointers refer to.

The bits of a structure containing a string don't also include the contents of the string itself.

"any array or struct containing one of the exceptions" is already talking about comparing the contents instead of the types and pointers. I didn't bring up the idea, I just said strings did the same, and with everything doing the same thing it's not an exception to bitwise comparisons, it's a general rule for how to handle nesting.

Floats are still an exception, but not the rest of that list.

That is not really what's going on here. If you replaced the interface in the OP's example with the actual type being returned, the nil comparison would work as expected. It isn't that the interface is allowing a comparison that would otherwise cause a build error — the interface is creating an error that wouldn't exist otherwise.

It's really not confusing if you understand the type system (yes, yes, I know).

A nil pointer is a valid value and can implement methods and satisfy an interface. An uninitialized interface cannot.

  type foo struct {}

  func (*foo) String() string { return "foo" }

  type Stringer interface{ String() string }

  func main() {
   var x Stringer
   x.String() // panics
   x == nil // This is a nil interface value
   x = (*foo)(nil)
   x.String() // works
   x != nil // This is not a nil interface.  It is set to a valid implementation, even if it's implemented by a nil pointer.
  }

From TFA, this was a fascinating comment. I'd never heard of V before, but definitely interested:

https://bitmaybewise.substack.com/p/when-nil-is-not-nil/comm...

Yeah, that sucks, but I can live with it. Generics can help avoid the problem (in a few places).

I'm not fond of the proposal. Or rather, I'd prefer priority for non-nullable types. Skirts around the problem all together.

This is the reason why I stopped learning Go.

https://www.youtube.com/watch?v=OdNQt4a6f7g

    How do I know my youth is all spent?
    My get up and go has got up and went
    But in spite of it all, I'm able to grin
    When I think of the places my get up has been

-Get Up and Go, Pete Seeger

please stop using substack or medium or anything with a paywall.

You object to people writing things and trying to charge for them?

They can do that but if it's a real paywall then it's not very appropriate for linking on HN.

Substack doesn't have a paywall.

Substack authors can have paid content, this is categorically different.

Anything an author marks as free, anyone can read. Substack doesn't have separate subscriptions for Substack, there's no "you've read your Nth free article!" etc.

Conversely, if something is paid, you have to pay for it.

Consequentially, if you see a Substack link here, you will be able to read it.

You are right, but Substack nevertheless presents a screen that looks like a paywall even on free articles like this. You need to notice and click a small "read the post first" link below the faux paywall into order to continue reading the post.

Yeah that's a bit nudgy, if you will, not my favorite thing either.

I see that more like big popup "PLEASE SIGN UP FOR EMAIL, THEY HAVE MY KIDS" overlays with the tiny X in one corner. It's annoying, the designer should probably be shoved in a locker, but it's not a paywall.

Reply on news.ycombinator.com