Robust Clojure: The best way to handle nil
bsima.me
bsima.me
- validate agressively at the system boundaries (IO- or user- facing stuff)
- inside the system, clojure implementation code isn't particularly nil-aware
- inside each namespace, the most important functions (typically their public part) are covered with pre/post conditions. This checks/documents the arguments/return type + (implicitly) non-nilness.
- pre/post conditions are implicitly exercised with integration tests.
There's clojure.spec/fdef if you prefer that to :pre/:post.
Say you're making an application that tracks corporate fraud and one of the changes that you need to make is to include the date of incorporation onto the corporation model.
For the existing corporations in your database what do you do?
What I see people do in the wild is to use a value like 1900-01-01. Or they make a ton of tables so that the nil is now just a non-existent key instead of a non-existent value.
I dislike nil as well, and I try to avoid it if possible, but I can't understand how to handle certain situations without it or without re-creating it.
I use an explicit value to represent the absence of something specific like :id-unknown which is much more useful and safer than just throwing nil in there when a project scales.
If something is nil in Clojure, you have to ask why is it nil? The answer is not always simple. It could be a bug, it could be legit missing data, a number of things.
A simple typo in Clojure can give you nil. You don’t want to interpret that the wrong way.
This (anti-)pattern has a Wikipedia page, complete with Criticism section.
https://en.wikipedia.org/w/index.php?title=Null_object_patte...
Incidentally, Objective-C does something like this out-of-the-box. Unlike Java, where invoking a method on null causes an exception to be thrown, or C++, where you just get good old undefined behaviour, Objective-C 'defaults' to doing nothing. This is described in the Wikipedia article. Strikes me as a very bad idea.
At the risk of just re-stating myself:
In Java, if you attempt to invoke a (non-static) method on a null reference, it throws a `NullPointerException`. This is exactly what you want: it's immediately obvious to the developer, and to a user. No further damage can occur as a result of the bug. (Ignoring broader questions of exception-handling, of course.)
Java pretty consistently adopts this philosophy of runtime checks everywhere, even on production builds. C#/.Net does the same. It's a valuable feature for software correctness.
(Java's choice of exception name is curious given that Java has references and not pointers, but still, they have the right idea.)
C++ is the opposite: if you invoke a non-static method (or 'member-function') on `null`, you get 'undefined behaviour'. Anything could happen. Hopefully the program will crash with something akin to a 'segfault', but maybe it will do something disastrous.
Unlike Java, the C++ philosophy is generally performance first, and it does few runtime checks for you. C++ has its reasons, but this approach is extremely unhelpful for software correctness; it can make both bug-detection, and bug-hunting, much harder.
(Incidentally, GCC and Clang feature a `-fsanitize=null` flag which gives you Java-style runtime checks, presumably with a moderate runtime performance cost. This isn't in the C++ standard, though.)
Objective-C is somewhere in the middle: you don't get undefined behaviour, but neither does the system tell you when you've got a null-dereference bug. It just silently carries on, and you're left assuming that your code is working fine.
There's nothing at all special about GUI code. Fail-fast is still what you want. There could be any number of technical or non-technical reasons you've found OS X GUI applications to be more stable. Null handling is just a tiny piece of the picture.
And the program often keeps running and generally working despite some potentially rare-null related bug occurs.
For a contrived example, imagine a user tries to paste a text field into a table and the text has an emoji in it. Let's further imagine the code doesn't handle full unicode properly, and the internal function handler returns a NULL, breaking the data model. In the die-on-null case the entire app often dies since the devs didn't think to add a try/catch around this case, thinking user strings from the UI object could never be null. Now the user has lost an entire table of data they've entered and don't know why for sure. The app just died when they pasted a bunch of data.
In the case of Obj-C with "ignore-null" behavior the data model would effectively ignore the field and likely result in an empty field. The user would still be able to save the rest of their data, with the curious exception of this field. They try pasting just that field and realize it has an emoji. Maybe they'd file a bug that emoji's don't work but they work around and enter a text smiley face. Of course many other things can/could happen like weird data corruption. But I've experienced very similar occurrences in apps and worked around them without data corruption.
Now as a developer, it's harder to find the reason as you don't have a nice log with "app crashed on NPE", but your users can still find a work around and often still find the app useable.
This scenario still requires defensive programming around sub-system interfaces to prevent said errors from reaching, say the internal embedded db, though that's already usually the case in a well designed code base.
> There's nothing at all special about GUI code. Fail-fast is still what you want. There could be any number of technical or non-technical reasons you've found OS X GUI applications to be more stable. Null handling is just a tiny piece of the picture.
For general systems code, I'd agree with you on null detection and fail-fast behavior. As a developer it's much better to know sooner when an error occurs. Granted null handling is just a small component of an overall system, it is an important component and NPE's are some of the most common errors and are most likely to cause an entire app to die due to unhandled/unexpected nulls.
However, from a simple analysis as an end-user of various GUI systems I prefer that some random probably unused feature silently fails or does weird stuff but that the main program keeps running, rather than an entire application (or worse the whole DE) fail due to an un-handled NPE.
And I disagree that there isn't anything unique about GUI code. It's a very different domain than systems / server applications. It should therefore utilize coding patterns which produce better results for that target domain.
[EDIT: Emphasis]
Or so I understand
Programs DO things, use what you're doing to describe your program, whenever possible.
In a better world, max([]) # -> None(T)
But sometimes we have to get creative...
assoc {} (assoc nil :a 10) => {:a 10}
concat '() (concat nil '(a b c)) => (a b c)
str "" (str "abc" nil) => "abc"
comp identity ((comp nil inc) 42) => NPE
+ 0 (+ 1 nil) => NPE
* 1 (* 1 nil) => NPE
Though, lots of this is inherited from common lisp where the list operations have more to do with lambda calc and the physical construction of the old lisp machines than they do anything else.
What Haskell offers is _generalisation of abstraction_. So, the function that satisfies the behaviour of fnil is called fmap, but it’s also the function that maps a function over all the values of a list, or the return values of a function. Equally, the equivalent of some->> (do notation) works for all of the above and more.
None of this requires you to know what a Kliesi category is.
So while you don't need to know anything about abstract algebras to do Haskell, that knowledge is lurking so close to the surface that it is really more of an issue than is generally acknowledged.
What Haskell offers is _generalisation of abstraction_. So, the function that satisfies the behaviour of fnil is called fmap, but it’s also the function that maps a function over all the values of a list, or the return values of a function. Equally, the equivalent of some->> (do notation) works for all of the above and more.
None of this requires you to know what a Kliesi category is.
These three paragraphs are unintentionally enigmatic and emblematic! Literally, the only thing I understood in the 2nd and 3rd paragraphs was, "What Haskell offers is generalization of abstraction." It's as if some large fraction of the Haskell community is significantly lacking self-awareness of how its own PR comes across. Been there and done that, just with Objects.
EDIT: Which is to say, A community that engages in such lack of self-awareness is dooming itself to hipster obscurity.
Let’s say we have a function addOne. Then
> fmap addOne Nothing
Nothing
> fmap addOne (Just 2)
Just 3
> fmap addOne [1,2,3]
[2,3,4]
> fmap addOne parseInt
(A function that reads a string for an int and adds one to the result.)
So fmap is polymorphic over a whole bunch of things that are unrelated in most other languages. (You might be asking why it’s called fmap and not map, and I wouldn’t have a good answer for you.) So, the abstractions of “Nullable, List and Function Return Value” are all part of a more general abstraction, Functor.
Now, at this point some people try to look up what a category theoretic definition of a Functor is. That’s fine, more power to their elbow. However, really all a Functor is in a practical sense for a dumb programmer like me is _something that supports fmap_.
My point is, it’s this stuff that’s useful and cool in Haskell. It won’t help you understand whatever they’re talking about on r/Haskell. And my only point about Kliesi categories is that you don’t need to know what they are. There’s a million and one interesting computer science concepts that are usable with a type system with Haskell’s properties, but you don’t have to use them all.
Category theory is really just useful because it shows you the building blocks. Some functors are not able to be used as monads. It's useful to have some intuition as to why. Even some things like applicative (a functor with a function in which you can successively apply multiple parameters) pop out and give you some useful insight (non-applicative functors can't be monads -- it's not obvious why until you try to implement it).
There is value in this organisation. The words are not fantastic, but again, they are just words.
I'm pretty sure I could get a student to write useful Haskell code in that time span with just mentoring and existing documentation material.
With Clojure, we're able to do a very quick ramp up, and then have new hires write code with very little assistance from the rest of the team. I simply haven't seen this be the case with Haskell even for experienced developers.
The people I mentored, though, could clarify any misunderstanding and get explanations from multiple viewpoints from me -- after years of experience with these abstractions. With such mentoring, you don't have to go through all the confusion phases.
If Haskell works for your team that's great though, it did not work for mine.
We don’t all have to be at EK, BM or GGs level.
Unless Haskell drastically fixed its record syntax recently, this is definitely not the case.
import Control.Lens
data MyRecord = MyRecord { _a :: Int, _b :: MyRecord }
makeLenses ''MyRecord
over (b . b . a) (+5) myRecordhttps://raw.githubusercontent.com/target/row-types/master/ex...
That's exactly what I don't like about some languages, if everything is a function, then it's all a big ball of mud. The only thing you can do with a function is call it.
I'd rather have classes of capabilities. Some things are callable, others are iterable, some are printable. But if it's truely about abstraction generalization, that sounds like a mess.
Not really, each distinct function will still have a distinct type. (This is largely why I prefer ml-family languages over lisps)
>The only thing you can do with a function is call it. You can also abstract over it, and pass it around. Which allows you to build whatever you want, numbers, booleans, if-then-else, etc.
The combination of function as your basic unit of abstraction and types as the differentiating descriptor is kind of the opposite of a mess, as you have a correspondence to familiar logic operations.
A function type A -> B is implication (given A, we have B), a product type (A,B) is conjuction (A and B), a sum type A|B is disjunction (A or B). (Sure, the logic isn’t necessarily sound, but it’s still useful).
When you have the fundamentals down you can build whatever capability system you need on top of solid abstractions.
I’m currently working on an ETL-project in haskell and it’s structured around a similar capabilities divide as you describe, defined by typeclasses/interfaces, it’s just all functions.
>The only thing you can do with a function is call it.
You can also abstract over it, and pass it around. Which allows you to build whatever you want, numbers, booleans, if-then-else, etc.)
(or x 5) ;;=> returns x if x is not nil, else 5
(-> {:a 1 :b 2}
:c
(or 5.5)
inc) (-> {:a 1 :b 2}
(:c 5.5)
inc)Should this be:
> ;;=> returns x if x is not nil, else 5
`(some-> m :c (inc 5))`
there is no need to defend against nil values. Hickey has your back on built in functions.
At the same time, I've found Haskell to be too mind-bending to use for any but the most trivial of tasks. (Eg. monad pyramids and the most cryptic error messages in the history of CS!)
Fortunately I've found a language that covers all bases: OCaml. It has static typing with full inference; algebraic datatypes; eager evaluation by default (yes, I believe Haskell's default of lazy to be counter-productive); no null / nil value; fast compilation to machine code and to Javascript; a compact syntax; and good editor support, including a modern gofmt-inspired auto-formatter.
This caused an issue when I was using (fnil + 0 0) in a reduce function - as it would barf on an empty collection - I got around this by providing an init value to the reduce call, but it caught me off guard when (reduce + []) works
I'm curious: Do any of you use the Cats library? Do you like it?
Therefore, I prefer to hoist missing value handling up to the caller, akin to "smart vs dumb components", where the caller has sufficient context and control to recover from errors. For example,
Instead of:
(defonce bus (ZeroMQ. "some-url")) ;; this might be null, but it'll leak into helpers that should only care about values
(defn transmit! [bus payload]
(if bus ;; why???
(if payload) ;; gross
(.send bus (zmq/encode-string payload))
(log/error "nil payload whoops")
(log/error "something bad happens that the caller doesn't understand and can't do anything about")
(defn tick! [ms]
;; athe wrong place to handle initialization problems
(if-let [input (.read bus buffer-length)] ;; multiple failure modes
(transmit! bus [:some (inc input)])
(comment "do nothing")))
(defn transmit! [^ZeroMQ bus payload] ;; type hint
{:pre [bus payload]} ;; basic guards
(->> payload ;; notice lack of nil-checking. you have to enforce contracts somewhere
(zmq/encode-string)
(.send bus)))
(defn init! [{:as config :keys [zmq-url]}]
{:bus (ZeroMq. zmq-url)
:redis ...
:logger ...})
(defn tick! [{:as system :keys [bus]} ms]
(if-let [parsed (parse-input (.read bus buffer-length))]
(transmit! bus [:some (inc parsed)]) ;; parse-input always returns valid data
(log/error "it's fine to handle runtime parsing errors here"))
(defonce !running (atom true))
(defn -main []
(let [system (init! {:some "config-url"})] ;; attempt bus failover here
(if-let [bus (:bus system)]
(go-loop []
(if @!running?)
(do
(tick! (time/now))
(Thread/sleep 30)
(recur))
(log/info "Shutting down..."))
(log/error "Retrying serial bus connection... ."))))My central point is that it is not reasonable to expect the consumer of a value to recover from a failure mode that was generated at the producer. Rather, it is better to handle failover and retries at the producer. Of course, once a (valid) value has left your system, any downstream errors are the purvue of the subsystem, but you may want to provide a signalling pathway back to the producer if it can be resolved upstream.
This naturally results in writing thinner consuming functions and "fatter producers" or "smarter callers."[^1]
https://medium.com/@dan_abramov/smart-and-dumb-components-7c...
Cixl [0] doesn't derive nil from every other type, instead Nil and most other types derive Opt; which means that user code may specify Opt to accept nil or any other type to have them automatically trapped by function dispatch. I find this to be a nice compromise between wrapped optionals and shooting from the hip.
The whole problem with Clojure, you have to remember how to write anything instead being restricted to a subset of a safe idiom.
The article is good and the author is right, but it should not be the author's problem in the first place.
Couple it with Expound[1] to make reading the spec errors a bit easier, and you've got a much better story for handling nil in Clojure.
[0]https://github.com/jeaye/orchestra [1]https://github.com/bhb/expound
Isn't that already what happens when you use the map as a function directly? Even the sub-feature of being able to provide a default value if the key is not found is supported.
> Over at Lispcast, Eric Normand argues for the “nil-punning” approach, which is fine. But I think this approach requires a confused notion of what nil/Nothing actually means. […] It is much simpler to understand nil as Nothing, i.e. the absence of a value (which is a type).
Assuming that, the behaviour of `get` is specifically one you do not want.
Furthermore, calling the map actually behaves as "Haskell and other ML-ish languages". `Data.Map.lookup` takes a Map, not a Maybe Map.
Responding to your edit: it’s safe to do it in Haskell because you know it isn’t nil at compile time.
It's no more fragile than any other function which doesn't do nil punning, and again the essay we're supposedly discussing advocates not using nil punning.
Using get is ambiguous, getting a nil result may mean that the key was not found or that there was no map in the first place.
(gethash key hash-table)
My personal ranking would be along the lines of:
1. Option types, that only makes sense for statically typed languages but it's the most clear and helpful and all other styles can trivially be composed from that.
2. Erlang-style, that's less the API which is similar to (though less forgiving/error-prone than) CL/Go and more the language: find/2 returns `{ok, Value} | error`, so to actually get the value you must either assert that the retrieval succeeded by pattern-matching on `{ok, Value}` directly (faulting if that didn't work) or properly pattern-match on both cases, and it doesn't give the illusion of a value on failure. Erlang also provides (3) and (5) via get/2 and get/3.
3. Python-style, the default is to raise on missing keys which makes it unmissable that the value was not there, alternatively provides a variant of (5) which can fairly easily be used as a (4) via the get method (return a placeholder value which can't have existed in the map, check against it by identity).
4. Common Lisp & Go, both will implicitly ignore the presence flag by default (bind result to single value) and just return a default on miss in that case. Common Lisp has an edge because Go's statically typed so it could make better use of MRV, and CL has much better abilities to manipulate and reify MRVs (only thing you can do in Go is bind them).
5. Clojure, Ruby and (finally) Java >= 8: a missing key returns a valid default[0], but it's also possible to customise that default (oddly enough in Ruby the same method which lets you get a non-hash-global default also lets you opt into (3), but I don't think it's used much?). For Clojure and Ruby this lets you emulate a (4) the same way Python does.
6. Java < 8/JS: a missing key returns a valid default (null/undefined) and there's no way to differentiate "there was no value" and "this was a value" save by a second access (in/contains).
[0] as in the default they return could have been a valid value all along
Also, since MRVs are a language construct, you can abstract over them to implement the equivalent of Haskell's >>= / maybe or to throw on a missing key. So, I'm not really sure there's a decisive advantage of Options vs. the MRV solution. In fact, if you change your perspective a bit, Options (and possibly sum types in general) are just ways of getting multiple return values in languages that only support single return values: i.e. you could think of the case of the sum type (i.e. "Some" or "None" being one of the return values and the wrapped value(s) being another.
Checking what needs to be checked is not boilerplate.
> If you care whether or not the key was present, the second return value is there and, otherwise, you just continue on your way.
Which is the entire issue. By default, if you're not careful or don't check, you just get garbage without warning.
> Also, since MRVs are a language construct, you can abstract over them
No, MRV being a language construct does not allow that at all (you can't do any such thing in Go because MRVs are language structures rather than properly reified). You can abstract over MRVs in CL despite them being MRVs, because CL specifically provides a bunch of dedicated constructs for working with MRVs.
> So, I'm not really sure there's a decisive advantage of Options vs. the MRV solution.
That options force the developer to deal with the issue and do so by default.
> In fact, if you change your perspective a bit, Options (and possibly sum types in general) are just ways of getting multiple return values in languages that only support single return values
Utter nonsense, MRVs are products. The entire point of sum types is that they're not, their purpose is to provide alternatives in a type-safe manner. The way to get multiple return values in "languages that only support single return values" is tuples.
> you could think of the case of the sum type (i.e. "Some" or "None" being one of the return values and the wrapped value(s) being another.
You really could not, that's the entire point.
Oops, I meant "because MRVs are a first-class language construct in CL (if only because of macros), you can abstract over them ..."
Secondly, tagged unions are a very standard way of simulating sum types in languages that lack them. All I was saying is that MRVs can be used in a similar way. E.g., an option can be replaced by a pair of values, one of which indicates whether or not the value is present and the other indicates what the value was.
As far as the boilerplate issue goes, if you really need to make sure that a field is present, you should be using hash tables in CL, you use a class and when you try to access an unbound slot, it throws.
Finally, it's just wrong to say that Options are only useful in statically typed languages: any _strongly_ typed language can use them effectively, whether or not those types are checked at run time or compile time.
case getSomeData of Nothing -> Nothing Just a -> a
Unless there's something about a that I don't know.
edit: replaced option with maybe.
The roach motel aspect of monads is not underestimating.
Did the world just completely forget about Null Object (design pattern)?
With a JIT, the calls often get converted to NOPs. Makes walking a graph clean, fast, trivial.
You presented a solution, not the solution.