NIL | Hacker News Reader

NIL

lispworks.com

154 pointssusam59 comments

This is what I consider to be a wart of Common Lisp: somewhat unsightly, not the most desirable thing to have, but rarely if ever impeding on your ability to do anything. Common Lisp as a language “knows” that NIL can be troublesome, so the rest of the language typically gives you facilities to avoid its pitfalls. Examples:

- GETHASH returns two values: the element found (or nil), and whether it found the element or not.

- GETF (which extracts a value from a property list) has a third optional argument for the value to return of the property isn’t found. It’s easy to make a unique sentinel value if you’re worried about collisions, for instance with an uninterned symbol:

    (let* ((not-found '#:not-found)
           (x (getf props 'key not-found)))
      (unless (eq x not-found)
         ;; do something only if KEY was found
         ))

In this case, it’s not possible for PROPS to ever contain the NOT-FOUND value.

NIL is one of the things Lisp had to compromise on to truly make it a “Common” Lisp.

The issues with nil aren't return values from containers. That problem exists regardless of what object you return for representing "not found", since that object could be a legitimate value in the container. It is an inherent problem of in-band signaling, and the only fix is out-of-band signaling: multiple API calls, multiple values, wrapped values (e.g. returning a hash entry object containing a key and value), or alternative control flows (e.g. not-found exception).

Now there is a real issues with nil being a symbol, Boolean false and empty list, and it is this:

In a heterogeneous, recursively varying data structure, there are are possible ambiguities as to what nil represents:

- is it a recursive element of the nested list structure: an empty sub-list?

- is it just a symbolic element, so that we consider (a b nil t) to be a list of four symbols?

- does it indicate false?

In many cases, this problem does not exist for various possible reasons:

- the data structure does not use multiple representations. Thus if nil is seen, it is known that it is a symbol, Boolean false or empty lists.

- the data structure does use nil in multiple roles, but they are disambiguated by syntax: such as fixed positions in syntactic nodes, or other conventions.

However, even if the problem is nonexistent in a given scenario, the data structure itself doesn't make it clear; there is still a possible readability problem for someone not familiar with the conventions. Someone staring at the data structure still has to guess about what is the meaning of a given nil, and refer to the documentation or, failing that, code.

Furthermore, the system itself has a "printability problem". The Lisp printer has to decide whether to render nil as nil or as (). Sometimes you see suboptimal choices like the system emiting (LAMBDA NIL (FOO)) where you'd like to see (LAMBDA () (FOO)).

This is the price we pay for the conveniences brought about by nil.

It's oh so convenient until it isn't; until you need to make distinctions between missing values and booleans.

I still haven't decided if it's a good idea or not, and I've been hacking Common Lisp for more than 20 years.

In which data structures? For example in CL hashes use multiple returns so in 99% of cases you can just not care, and when you do care about the difference you just check the second return value.

Yes, I think the two return values convention in CL mostly makes this a non-issue.

I've run into issues when interfacing databases and writing interpreters, and probably more that I've forgotten.

They're just minor culture clashes that add up...

I like the way CL does things but it makes dealing with JSON occasionally annoying.

Do you mean, for example, a null node versus a missing node?

I mean distinguishing between `null`, `false`, and `[]` on the JSON side. Most JSON libraries for CL by default parse them all as `nil`. (ST-JSON excepted.)

I guess that means you can't generate false and [] either?

So this would be a good example the culture clashes I mentioned, handled badly.

In general, even if you do have a separate boolean type, you need to distinguish between the function which intended to return null and the function which was unable to return anything.

We see here also the distinction between sum types and union types: Maybe Maybe a has three variants: None, Some None, and Some Some a; but (T | null) | null collapses to just T | null.

This has a simple solution. In the same way that Maybe is effectively a container, use a container for the Lisp value. I.e., a list.

    None -> '() 
    Some None -> '(nil)
    Some Some a -> '(a)

Yes, and this is what e.g. assoc and member do. But in those cases there's already a container, whereas in a hash table you would have to construct a new one. I do think that multiple values are easier to use and result in more direct code; the problem is that they don't enforce correct usage the way a container does.

SQL has true, false, and null, for a good reason.

The nil / null value is an attempt to cheaply produce from type T a type like Maybe T, or Optional<T>, or basically T | nil. It's sort of practical. OTOH making false and nil indistinguishable is much less convenient than one might think, and leads to subtle errors.

You're saying I should use custom symbols like 'true and 'false and put in checks like (eq 'false x) instead of (null x)? To avoid subtle errors? Honest question.

Postmodern (https://quickref.common-lisp.net/postmodern.htm), a library to interface with Postresql will use the keyword `:null` to represent null, and T and nil to represent boolean true and false.

And then SQL implementations use empty string as null :)

JS has false, null, and undefined... and NaN and 0... and nobody is ever confused.

I have been using Common Lisp since about 1982 and it is probably my favorite language both because I am used to it and because old code usually runs forever without modification. That said, I now also often use Racket and Swift that both seem fresher, but don't have the extreme stability of CL. I often use Common Lisp libraries that haven't been touched by the creators for 5 or 10 years (or more).

I once wrote up a little rant about future software being ancient, with no changes for centuries. What might the far future be like? I was imagining a world where security and robustness were considered much more valuable than a never ending flow of new features. Could popular programs developed over many centuries evolve to a point where it made no sense to modify them?

I really like this perspective. One of the more, well, common, criticisms of Common Lisp is that the standard is stagnant. Reframing it as stable is great rhetoric and also sound dialectic. I've only been using CL for about half as long as you have, but I still appreciate that stability greatly. Also, it's not as if CL deprives neophiles of opportunity[1].

[1] https://lisp-journey.gitlab.io/blog/state-of-the-common-lisp...

I remember running into nil a lot when trying to use Clojure. My immediate sensibility was very "yuck". It allows for nice chaining of APIs but it seems to really just mess up the notion of optionality so much that I'm curious about how often it's a bug generator.

I understand that the sort of spaghetti-code from constant nil checking would be painful, but I can also imagine a lot of hidden bugs when it comes to dealing with things like user-provided data. Would love to hear the experience from Lispers who also have experience with "null"-y issues in other languages.

Posted under another comment, but I've personally experienced it being a big problem where a Lisp interfaced with another language: https://news.ycombinator.com/item?id=27878200

Clojure doesn't have this problem, in Clojure nil is not the empty list, and the empty list is not nil.

    ;; Nil and empty list are different
    (nil? '())
    ;> false

    ;; Empty list is not falsy
    (if '() true false)
    ;> true

    ;; Only nil is falsy
    (if nil true false)
    ;> false

    ;; Prints as empty list
    (pr '())
    ;> ()

Good decision on their part. Unfortunately ours was CL-inspired

I've got professional experience with null in Java, C#, and C++, as well as Clojure. From my experience "nil" in Clojure is the least "issue creating" of the bunch.

Generally when "nil" is an issue in Clojure, it's actually from the Java side, where Java doesn't "do the right thing" for "nil" and just throws a "null pointer exception" instead.

In Clojure it's mostly a problem when it hides that you just aren't getting something because of a bug, as opposed to because it truly isn't there.

The most common one for me being map lookups:

    (get m :fooo)
    ;> nil

Now everything after will "work", because everything handles "nil" input and generally does "the right thing", like treat it as false, or returns nil if given nil, or treats it as identity, etc. But you actually just messed up the key and that's why you are getting "nil", where you wanted to do:

    (get m :foo)
    ;> "foo"

I'd say this is the most annoying case for "nil" for me in Clojure.

Agreed. The problem often presents itself downstream, or in a totally different place from where it originated.

This is exactly the 'null' bug, and the problem that's solved with a strongly-typed language and an `Optional<T>` or `Maybe T`.

Then you are forced to deal with the fact that you might not have a value, right away.

Optional<T> actually has the same issue I think. Clojure properly handles "nil", the problem is not that you need something to force you to deal with not having a value right away, the problem is that Clojure automatically deals with the value not having a value right away for you.

Imagine you have this:

    Optional<String> get(String k)
    {
        ...
    }

    get("fooo").orElse("not found");

This is the same bug, the bug being it's normal that "fooo" is missing, you should have written "foo" which is the real place where the data is supposed to be.

Or imagine for any other reason `get` returns an Optional.empty() when it shouldn't.

It's kind of the flip side of "null". Normally nulls are bad because of nullPointerException and people forgetting to handle null. Clojure in general handles null everywhere, and so you won't get nullPointerException, instead things will just work with whatever defaults makes sense for everything that comes after and is getting the null.

That makes it harder to realize that the null is there not because it missing, but because you have a bug which makes it look like something is missing when it should be there.

  (get m :foo :not-found)

Problem solved.

I wouldn't say this totally solves it:

    (get m :fooo :not-found)

Is still a bug that'll be hard to debug, because your code will most likely handle :not-found and believe that it is truly not-found for some reason, but it's not, you messed up the keyword.

Now you're getting at a much more fundamental issue. The issue here is that m is not a product type but an open map that allows for an arbitrary collection of keys. No static typing system will solve this either.

If you're really that worried about typos and don't need an open map, Clojure has a solution as well in the form of defrecord and deftype. Now the compiler will catch the problem.

Yes, I don't really think any language solves this problem, just moves it around.

It's just an issue in general whenever the happy path includes optionality as a possible result, how do you then distinguish the happy path of getting a missing value, which you expect is a possibility given the use case, versus a bug to have set the value or have been provided the value or have queried for the value?

I think the only languages that reduce the scope of this problem are those where you can define non-nullable types. In that at least when you know for a fact your use case shouldn't have "missing" things, the compiler can catch those bugs. You can also handle this in Clojure with runtime assertions, albeit catching the bug a little later in development.

But for the times when optional is a real possibility, you still have this issue.

If you move to an explicit set of valid keys, it could be that something failed to assign them a value, etc.

Edit: I guess maybe something where the language simply has no null ever could work. Like say a field not yet assigned a value would have the value :unset, and so if someone doesn't provide a value for the field, you'd still set it, but with the new value :missing. Now technically you could distinguish a bug as `:unset` versus actually missing as `:missing`. I'm not sure this would work for dynamic use cases though, where like you don't know the full range of possibilities in advance. But if you do, you could check that nothing is `:unset` and if it is, you know there is a bug, otherwise it should be `:missing` or some other value.

Edit2: Ok, now that I played a bit more with what you said, I realized thay your solution is actually better then I thought, but it depends how you model things.

    (get {:foo :missing} :fooo :unset)
    ;> :unset

    (get {:foo :missing} :foo :unset)
    :> :missing

If you model optionality as an explicit `:missing` value on a set key, like assoc the key `:foo` with value `:missing` to indicate that no value was provided for it. With the 3rd argument to `get`, you can now tell the difference between a bug or an actually missing value.

It does seem to require a bit of care, and I don't know if its just.moving the issue somewhere else, but it could be an approach.

Here's the realization that got me over the "yuck" of nil punning, well two realizations.

First, the whole billion dollar mistake is nil POINTERS, not the value nil. Nil in lisp is a value. This is not really a problem, and in fact there's even a design pattern of the null object pattern (https://en.wikipedia.org/wiki/Null_object_pattern) to reproduce this behavior in OOP languages.

Second, in CL, there really is no such thing as nil. Nil is the empty list, they are the same thing. you could search/replace every instance of nil in your code with '() and you'd have the same code. And before you get flashbacks of Javascript and null, just remember unlike JS, there's only a single falsey value and that is the empty list, and there is no automatic type conversion that has surprising boolean semantics.

Ultimately CL manages to turn nil punning into a very nice feature that makes the spaghetti code you're talking about unnessary while also avoiding many of the 'null'-y issues of other languages due to

1) nil is a value

2) there is only one falsey value and that is nil

3) no automatic type conversion

I've been writing a bunch of Common Lisp recently, where I usually have been more interested in Schemes. I'm less bothered by its little inconsistencies, like the empty list not needing to be quoted, than I used to be. Maybe because I'm getting older, and people develop cruft just like programming languages. I'm not really small and elegant anymore, so why should my Lisp be?

> I'm less bothered by its little inconsistencies, like the empty list not needing to be quoted, than I used to be

CL certainly has its warts, but that is not one of them. Most data types other than pairs and symbols are self-evaluating. You don't quote number literals, for instance!

That the list terminator is not self-evaluating is definitely a scheme wart. (Sadly, I'm told that motions to change this behaviour in r7rs-large were rejected.)

'(1 2 3)

'(1 2)

'(1)

() ; wart

That's syntax sugar and it's lying to you for convenience's sake.

  (+ . (1 . (2 . (3 . ()))))     #| --> 6       |#
  (quote . (1 . (2 . (3 . ())))) #| --> (1 2 3) |#

(1 2 3) is a pair and we have a rule for evaluating pairs: unless the car is a symbol denoting a special form (1 is not one such), apply the car to the cdr. So we need to quote it if we want to get it as a literal.

() is a special list terminator object; it is not a pair—as this dotted list syntax shows—and it has neither car nor cdr, so our previous rule for evaluation does not apply. However we have another conventional rule: objects which are not pairs and are not symbols should evaluate to themselves. This applies to strings, characters, numbers, vectors... Why not to the list terminator?

Never really thought that through. Thank you.

Perhaps this makes it clearer:

  '(1 . (2 . (3 . ())))
  '(1 . (2 . ()))
  '(1 . ())
  ()

The ' applies to the ., not to the ().

NIL: "New Implementation of LISP"

http://www.ai.sri.com/~delacaze/alu-site/alu/table/Lisp-Hist...

>The Macsyma group at MIT began a project during the late 1970's called the New Implementation of Lisp (NIL) for the VAX, which was headed by White. One of the stated goals of the NIL project was to fix many of the historic, but annoying, problems with Lisp while retaining significant compatibility with MacLisp. At about the same time, a research group at Stanford University and Lawrence Livermore National Laboratory headed by Richard P. Gabriel began the design of a Lisp to run on the S-1 Mark IIA supercomputer. S-1 Lisp, never completely functional, was the test bed for adapting advanced compiler techniques to Lisp implementation. Eventually the S-1 and NIL groups collaborated. For further information about the NIL project, see ``NIL---A Perspective.''

http://www.ai.sri.com/~delacaze/alu-site/alu/table/older.htm...

>``NIL---A Perspective,'' JonL White, Macsyma User's Conference, 1979.

https://en.wikipedia.org/wiki/NIL_(programming_language)

>New Implementation of LISP (NIL) is a programming language, a dialect of the language Lisp, developed at the Massachusetts Institute of Technology (MIT) during the 1970s, and intended to be the successor to the language Maclisp.[1] It is a 32-bit implementation,[2] and was in part a response to Digital Equipment Corporation's (DEC) VAX computer. The project was headed by Jon L White,[3] with a stated goal of maintaining compatibility with MacLisp while fixing many of its problems.

T (opposite of NIL) also existed <https://en.m.wikipedia.org/wiki/T_(programming_language)>

NIL is certainly an overly-overloaded concept in Common Lisp. Scheme did a better job by separating the boolean type from the empty list but in practice I don't find NIL much of a hindrance in CL.

Huh? In my opinion it’s Scheme’s one million dollar mistake. Common Lisp’s identification of NIL and the empty list makes things very ergonomic in so many cases that I wonder if it’s by design or we were just lucky.

NIL equating to the empty list is fine. The empty list equating to: didn’t find what you wanted, but here’s an empty list instead? Hope you weren’t looking for an empty list though, then you’d be really confused, sorry. Nah, I’ll take “false” thanks.

> Nah, I’ll take “false” thanks.

false is an object which can be stored in the container, and which you could be looking for. You've just migrated the problem away from empty lists.

That's right, then I'd just have a mirror of the original problem. I could store empty lists to represent false, but that's not really any better.

One place this is really problematic is serializing structures to JSON. I worked on a JS front-end at a company whose back-end was all Lisp, and I cannot count the number times the app broke with "TypeError: Cannot read property 'map' of null", because every single list that would normally serialize to an array would serialize to `null` when it was empty unless the person writing the service went out of their way to do otherwise.

At some point I wrote an API-validation utility almost entirely to help manage this problem.

An alternative solution is don’t serialize from/deserialize to NIL. Use MY-JSON:TRUE, MY-JSON:FALSE, MY-JSON:NULL, etc. to create a perfectly bijective map. Lisp is a symbolic language after all, no reason not to take advantage of it.

That would mean writing code for each endpoint to convert its response data into the JSON-explicit format instead of dumping it out directly; this wasn't a big company and we didn't have the man-hours for that extra layer

I think when they manually fixed it in individual cases, they did something like `(or the-list json:empty-array)`. We just had to encounter that case in practice and then make a Jira ticket for them to go in and patch over it, and then do that over again each and every time it came up :P

The utility I wrote[1] would match JSON responses against expected schemas and report the bad `null` (or otherwise), its location in the JSON, and which service it came from, in the JS console. It didn't stop errors from reaching production but it saved a lot of time debugging.

[1] https://github.com/brundonsmith/simple-typechecker

I dig the way this explicitly defines the null content of `nil`. It makes intuitive sense, but it also makes me more conscious of the general principle of NULLity.

related article on nil punning: https://lispcast.com/nil-punning/

I'm reminded of the fact that in Smalltalk, nil accepts any message, does nothing, and returns nil.

The same is true of Objective-C's object semantics, which means that language has two flavors of null pointer, one of which will scream SIGSEGV if you do anything significant with it, the other having more subtle failure modes...

You actually get message not understood and dropped into the Smalltalk debugger if you send a message to nil.

Just to be clear for @bitwize, nil is an object like everything else in smalltalk. It's an instance of UndefinedObject. It isn't special in any way and executes any method it understands just like all other objects.

Object>>isNil ^false

UndefinedObject>>isNil ^true

It drops into the debugger if you pass it a message it doesn't understand.

I think I might've described how Objective-C works. It's been a while since I worked in depth with Smalltalk.

https://software-lab.de/doc/ref.html#nilSym

NIL in PicoLisp.

Maybe having only one value of this kind is an effective way to make it really obvious which problems occur when you don't have an Optional type? Those problems also happen in languages with several null or false-like values, but you discover them too late.

NIHIL would have been more beautiful, but all programers are to lazy to even typ "Hi" so here we are.

I appreciate that the "to"/"too" typo could be part of the joke but know that it's still like sandpaper on the brain regardless.

Reply on news.ycombinator.com