Option and Null in Dynamic Languages
rntz.net
rntz.net
CL-USER 45 > (gethash 'foo (make-hash-table))
NIL
NIL
The first value is the stored value in the hash table. If there is no key, the first value is NIL. The second value indicates whether the key is present in the hash table.Another way of seeing the same thing is: Type-theoretically speaking, MRV is using a product type (two values) to emulate a sum type (one value of two possible forms); why not just use a sum type?
1> X = #{"a" => 1}.
#{"a" => 1}
2> maps:find("a", X).
{ok,1}
3> maps:find("x", X).
error
Constructing and destructuring tuples is cheap in the Erlang VM, so Erlang uses them where other languages would use types.Further, in idiomatic Erlang, you don’t bother with a branch for each case of the option type; you just do the following:
{ok, Val} = maps:find("a", X).
Effectively, that's a type assertion: if maps:find/1 responds with error instead of {ok, _}, that’s a process crash. (defn values-in [m ks]
(vals (select-keys m ks))) def values_in(keys, d):
ret = {}
for key in keys:
try:
ret[key] = d[key]
except KeyError:
pass
return ret
which works much more quickly in cases where keys is nearly all of d.keys, and doesn't force you to reinvent monads in Python, which is arguably unsuited for them.* Sanity values may vary for people who haven't gotten into the habit of using Python exceptions for little things.
from itertools import chain
def getOpt(d, k):
return [d[key]] if key in d else []
def values_in(keys, d):
return list(chain(*[ getOpt(d,key)
for key in keys ]))
Or Ruby: def values_in(keys, d)
keys.flat_map {|k| d.has_key?(k) ? [d[k]] : []}
end def values_in(keys, d):
return [x for key in keys for x in getOpt(d,key)]
Using lists as options is a neat trick I hadn't thought of when writing the article. Unfortunately, no dynamic language uses this idiom (or really any other non-exception option-like patterns) natively.Good point. For some reason, I have this giant python blind spot for comprehensions where the sources aren't orthogonal -- I miss them in the first pass all the time; I normally notice it fairly quickly in code I'm not posting on a discussion forum...
> Unfortunately, no dynamic language uses this idiom (or really any other non-exception option-like patterns) natively.
Yeah, to use it in your own code in most popular dynamic languages, you have to wrap library code that works differently. The good thing is that most popular dynamic libraries have good (largely functionally-inspired) tools for working with lists and other iterables that the payoff is pretty good. (Of course, with duck typing in most popular dynamic languages, one thing you could do to be more explicit is to combine the Option-as-class and Option-as-list approach, and make Option a class that implemented the interface used by immutable collections.)
// Using if/then/else
function foo($x) {
$results = look_up_something($x);
if (count($results) > 0) {
$val = $results[0];
return do_something($val);
}
}
// Same as above, but with a foreach loop
function foo($x) {
foreach (look_up_something($x) as $val) {
return do_something($val);
}
}
Of course, the nice thing about Options (other than being able to distinguish between "None", "Some None", "Some (Some None)", etc.) is the availability of combinators to lift and compose non-Option functions with optional functions. Sometimes we can map and fold ("reduce"), but sometimes we can't (eg. PHP's "array_map" doesn't work on iterable objects, which many APIs use in lieu of arrays).This seems more "Pythonic":
def lookup(d, key):
return Just(d[key]) if key in d else None
def values_in(keys, d):
values = []
for k in keys:
x = lookup(d, k)
if x:
values.append(x.value)
return values
If you really want a one-liner, I think this is easier to understand: def values_in(keys, d):
return [x.value for x in (lookup(d, k) for k in keys) if x]The obvious being said, you can also use functions and lambdas to get around having to create custom classes or abuse lists.
def value(x):
return lambda: x
def values_in(keys, d):
return [x() for x in (value(d[key]) if key in d else None for key in keys) if x]
def get_maybe(d, k):
return value(d[k]) if k in d else None
def values_maybe(keys, d):
""" If you want the caller to be responsible for handling the maybe case """
return [get_maybe(d, k) for k in keys] sub values_in(\@\%) {
my ($keys, $d) = @_;
grep { $d->{$_} } @$keys
} def values_in(dict, keys):
marker = object()
return [x
for x in (dict.get(k, marker)
for k in keys)
if x is not marker]Its not as composable as Option- or list-based approaches, though. (OTOH, you can use it to implement those approaches, and its cleaner for that than the way I did that elsewhere in the thread.)
Wait is he really trying to say, "Setting a VALUE to nil removes the entry from the dictionary!"? If so, that sounds terrible.
I can imagine barking up the wrong tree wondering why my map is missing fields when the real problem is an object I passed into it is unexpectedly nil.
//Let's define some object
var obj = {a: true, b: true, c: true, d: true, hasOwnProperty: true}
//Now let's make them falsy in different ways
obj.a = undefined; //set to undefined
obj.b = null; //set to null
obj.c = false; //set to false
delete obj.d; //delete key
delete obj.hasOwnProperty //delete key, but present in prototype
console.log(obj.a) //--> undefined
console.log(obj.b) //--> null
console.log(obj.c) //--> false
console.log(obj.d) //--> undefined
console.log(obj.hasOwnProperty) //--> [Function]
console.log('a' in obj) //--> true
console.log('b' in obj) //--> true
console.log('c' in obj) //--> true
console.log('d' in obj) //--> false
console.log('hasOwnProperty' in obj) //--> true
console.log(obj.a == null) //--> true
console.log(obj.b == null) //--> true
console.log(obj.c == null) //--> false
console.log(obj.d == null) //--> true
console.log(obj.hasOwnProperty == null) //--> false
for(var key in obj) {
console.log(key) //--> prints a, b, c
}
In Lua, the only data structure is a hashtable, which maps any non-nil value to any other non-nil value. It can also be set in the objects __index metamethod/table local tab = {a = true, b = true, c = true}
-- by default there are none, but we can defined a fallback metatable similar to a prototype method like js's obj.prototype.hasOwnProperty
setmetatable(tab, {__index={c='meta c', d='meta d'}})
-- if it is important that a key is explicitly set to false, do so, otherwise simply remove it
tab.a = false --set to false
tab.b = nil --set to nil
tab.c = nil --set to nil, present in fallback table
tab.d = false --set to false, present in fallback table
print(tab.a) --false
print(tab.b) --nil
print(tab.c) --meta c
print(tab.d) --false
The rules for Lua are simple: Look for the key in the table, if it's not there, check the fallback defined in the metatable, if not, then return nil.This is especially important in Lua, because any value can be a key, you would have to be sure to delete the key otherwise it could not be garbage collected. local tab1 = {1,2,3,4,5} local tab2 = {} tab2[tab1] = true for k,v in pairs(tab2) do tab[k] = nil end
If this worked like JavaScript, then tab2[tab1] would return nil, but tab1 could not be garbage collected until tab2 is. Because JavaScript objects can only have strings for keys, this is less of an issue.
http://www.oracle.com/technetwork/articles/java/java8-option...
So, sure, you got rid of null from the language. Does your program handle the case where it was handed a negative number correctly? All positive values? All unicode?
Consider, in the Ariane Rocket incident, they were using a dependently typed language. Didn't magically prevent errors in assembly.
So, fine, we'll make our programs where they handle all cases... Rapid prototyping probably just flew out the window. As did worrying about actual honest to goodness use cases.
A great quote by Knuth I saw once was where he acknowledged that tex would choke on mile wide documents. But who cares?
Aren't you just moving the goal post? You can play the 'OK so you can eliminate that from the equation, but can you ...' indefinitely. Well, perhaps only until you get to the question of cosmic rays.
> So, fine, we'll make our programs where they handle all cases... Rapid prototyping probably just flew out the window. As did worrying about actual honest to goodness use cases.
The topic is a few special cases, like null, not every conceivable case. If things like null and undefined cause the majority of crashes, and they are simple to handle with a 'proper' language (and they are), then you already have gotten a lot of bang for your buck.
For rapid prototyping: if handling null cases is too much, use a function that unwraps nullable values directly, and if they are null, crashes (and no, this is not the same as a null pointer exception). Then go back and clean up the program when you are not prototyping quite as rapidly anymore.
So, for example, a place where null comes up. When you create a file. Programs that didn't check just threw up on an exception now. It is just as likely that without that option, the code would have just checked for a value, or thrown an error. So... what changed?
(For the record, I'm assuming an interface such as:
fileOpen -> FileHandle
Where the FileHandle object can be null, and it being null is used to indicate that the call to fileOpen failed.)To be clear, this would involve the static type system for your language forcing certain things to happen, or not compiling. The sequence:
1. fileOpen returned a Maybe<FileObject>
2. You want to store the FileObject somewhere, and it expects an actual FileObject.
3. Maybe<FileObject> is not itself a FileObject, so you have to figure out: is it Nothing, or a FileObject?
4. You have to handle the Nothing case (language forces you to, even if you do nothing about it - you explicitly decided to do nothing).
5. If it is indeed a FileObject, you store it for later use, and you are guaranteed it is, indeed, a FileObject.
And I should note that I don't necessarily dislike the optional types and such. I just also don't have the major disdain for null that the internet seems to send through a massive echo chamber nightly.
That's a problem with Java (because it doesn't let you do that) and a similar problem exists with dynamic languages (because types aren't attributes of variables, so any variable could be null -- or any other value, null isn't particularly exceptional in dynamic languages for being a source of this kind of problem), but its not a problem "with languages that have null", as there are statically typed languages that have null but distinguish nullable and non-nullable types (C# and other static .NET languages are probably the most well-known examples, though there are others.)
> But really, at that point it's just sugar for an option type.
No, having a the free choice not use nullable types doesn't make the languages nullable types into sugar for option types. Nullable types don't compose the way option types do -- the issue in the article this thread is about isn't about whether or not you have the choice for nullable types, its about the fact they don't compose so you can distinguish a function returning a null as a substantive value because the thing you tried to store was also a nullable type and had a null value (Just None, in a world with Options instead of nullable types) or where the null is a result of the lookup failing (None, in a world with Options instead of nullable types.)
The deficiency of nullable types in C# compared to Options isn't the lack of choice of writing non-nullable types -- as you certainly have that choice for any data you need to represent -- its that nullabe types don't compose the way Options do.
Well, any value type -- but, yes, a custom type has to be a struct to not be nullable.
> How do you specify that a function can take a non-nullable string?
"string" in C# is the name of a particular reference (and hence, nullable) type. If you want a function that takes a non-nullable sequence of characters, that's quite possible, but the type of its argument won't be string. (The fact that there is no such type predefined and the entire .NET framework library is built around the assumption that if you want a sequence of characters you are going to use the built-in, reference -- and therefore nullable -- string type is an issue with the cost, in programmer-time, of doing this for any substantial use, but its not a "the language doesn't let you have a non-nullable type for any kind of data you want" issue.)
The disdain is probably (allow me to project) because it is a totally unnecessary problem which can be solved easily with virtually no drawbacks.
Granted, this can certainly be done with optional types. So, I'm not ultimately against them. I just think they are oversold quite a lot.
Laziness doesn't come into it, since you won't even manage to compile your code, let alone run it, if you try to use a "Maybe FileHandle" when a "FileHandle" is expected.
With that said, laziness does cause problems related to file handles, but those have nothing to do with unhandled failures. The problem is that files are a limited resource which we should release as soon as possible (just like memory, sockets, etc.). Lazy evaluation is at odds with this, since it essentially defers function calls to the last possible moment. One way of thinking about it is that laziness is "call-by-need", but we never really need the return values of functions like "closeFile".
Edited to add:
Note that this can be the case in Haskell when you're relying on exceptions for error handling. That's the opposite approach from option types, though.
So, depending on just how "lazy" the program truly is, sure you passed it something that unwraps a Maybe, but only once you actually go to use it. That make sense?
Consider this fun bit of nonsense scala
def foo(x:=>String): Unit = {
println("Hello")
println(x)
}
If I call this as "foo(None.get)", then it will print "Hello" before barfing. In fact, if I didn't evaluate x, it wouldn't even barf. I had thought the laziness of Haskel was that ++. (Meaning I thought if I stored x off to another also lazy reference, it wouldn't barf right away.)To make that a little more clear of what I meant, some even more amusing scala.
class Foo(x: => String) {
def happy = println("Doing good.")
def sad = println(x)
}
If I create something with "new Foo(None.get)", I can call happy on this object all day long. If I ever call sad, I get an exception.Apologies for the ninja edit above, but this scala seems to show it isn't just IO that is at danger here.
class Foo(x : =>String) {
def happy = println("I'm good.")
lazy val ack = x
def sad = println(ack)
}
Again, if I do "new Foo(None.get)", I can call happy as much as I want, but "sad" will blow up.Now, I do not and could not claim that this sort of nonsense would have proliferated with the same abandon as null. I am claiming that this is not that unlike a null pointer exception, though.
case maybe_string of
Just string -> newFoo string
Nothing -> defaultFoo
Or to use one of the relevant higher-order functions: fmap newFoo maybe_string :: Maybe Foo
maybe defaultFoo newFoo maybe_string :: Foohttp://research.microsoft.com/en-us/um/people/simonpj/papers...
This is also why the Control.Exception module exposes `evaluate`, which embeds a value in an IO computation and thus resolves this problem... If you remember to use it.
http://hackage.haskell.org/package/base-4.7.0.0/docs/Control...
Fortunately, in idiomatic Haskell exceptions are considered to be an expert feature (basically I see them in concurrent IO or resource management code only) and partial functions—usually introduced by incomplete pattern matching like `None.get` does—are veboten.
It's not perfect, but -Wall will help with that.
My point here was more that just because you have a file handle doesn't mean you have a valid place to write data. Making it optional would possibly prevent some bugs, true. However, it doesn't really help as soon as you have a filehandle to a full filesystem, for example. (Or, well, any other problem that usually happens to cause grief with the filesystems. You got a file, but by the time you went to use it the system was full and you couldn't, etc.)
Regardless, I should have been clearer on many points (and I fully accept I was flat out wrong on at least a few :) ). I do think null pointers are bad. I also know with proper static analysis tools, you don't need a whole new type system to make things better. Unless you are considering tools such as Coverity and friends some form of type system.
I would, for the record.
There was a thread some days ago (https://news.ycombinator.com/item?id=7916554) where I described some hackary I used to get particular guarantees out of C's type system. My response in https://news.ycombinator.com/item?id=7918423 seems tangentially relevant here. Type systems aren't (only/necessarily) features of a language.
I think the only other differences stem ultimately from standardization. Tooling could exploit any type system a program is known to follow. Likewise, compiler optimizations, &c, &c.
Well, I realize that is false. But the hoops all of the examples I have seen that types have to jump through to accomplish that aren't exactly pleasant.
So, maybe my preference is just that tooling can give you many of the benefits of more advanced type systems, without having to have all of the boilerplate of it everywhere in the code.
At any rate, I should say thank you very much for keeping this dialog going. Been a blast!
By "tooling" there, I wasn't referring to static analysis. I meant things like type-directed name resolution, type-directed completion, Agda's "holes", and so on. Strictly, these do not rely on the type system being "a part of" the language, but they do rely on some degree of standardization on a particular type system for the tool to work (or work well).
"Interleaved calls to different types, for example, can be found through tooling."
I actually don't understand exactly what you mean here. You might be surprised what even pretty rudimentary type systems can find if you leverage 'em right, though - as I mentioned in that other thread, I track which threads access what resources, and which functions live on which threads, with C's type system unadorned by additional checks (I do use additional static analysis, it just is entirely unnecessary to catch this one).
"Well, I realize that is false. But the hoops all of the examples I have seen that types have to jump through to accomplish that aren't exactly pleasant.
"So, maybe my preference is just that tooling can give you many of the benefits of more advanced type systems, without having to have all of the boilerplate of it everywhere in the code."
Ironically, I think my C code has more annotations for my static analysis tool-of-choice than the C types. Certainly there are a lot of poor type systems, and a lot of languages that expose particular type systems poorly. Boilerplate type annotations are mostly unnecessary in languages with type inference.
As I've said elsewhere, though, your tooling may be "more advanced type systems". Splint, for instance, can enforce linear typing - something that is not a part of the type system laid out in the C standard (or that enforced by typical C compilers, even with warnings).
"At any rate, I should say thank you very much for keeping this dialog going. Been a blast!"
Yes, it's turned out great :)
That is, at the point that a Maybe String statically changes to a String is not necessarily where the program will explode. A lot like a null assign. (Again, in a lazy language. Unless I'm mistaken on this subthread, the point was simply that you can still have that level of WTFery with Maybe that you can have with a null.)
Now, rereading the beginning, the reference was to a "Haskel like" type. So, I think I was definitely off the rails on this. I do feel I learned something going down this particular rabbit hole, though. Thanks!
There is a function, fromMaybe, which takes a Maybe x and gives you an x... but it also takes a default!
Often even more useful is the maybe function:
maybe :: output -> (input -> output) -> Maybe input -> output
This takes a default output to use if you're given Nothing, a function to apply to the input and generate an output if you're given Just an input, and Maybe the input, and gives you the resulting output. As I said this is often more useful than "and use this default", because quite often the value you want to generate when it's Nothing is outside the image of the input under the function in question. For instance: maybe "no number" show maybe_integer
There's no number I can pass to show to get "no number", maybe lets me very conveniently avoid trying to force one.Yes, they will. You could make the same statement about crime: "We'll never get rid of all crime, so why bother having police at all? Having police is just moving some goal posts." Does that make it clear enough how silly this sort of reasoning is?
This reads to me like eliminating null pointer bugs would just create a void that something else would fill? I don't see how that would work, at all. Is eliminating a null pointer exception going to introduce a divide by zero exception/bug? Is it going to introduce program logic bug? ...
If you're thinking about unwrapping a nullable value and throwing an exception if it really is null; I don't think of that as a conceptual null pointer exception, at least not in the usual sense. It's a deliberate decision along the lines of "I know [because of the control flow up to this point] that this value can not be null" or "if it is null, just throw an exception". It is NOT "this value can never be null... I think... well let's check the documentation, OK I'll assume that it isn't being misleading (documentation can easily 'lie' about a value being null in a nullable-by-default language, since non-nullable is not expressible in the language).
In a language with nullable-always, you can always pass a value (probably reference, but still) as null. In a language with explicitly nullable types, this is impossible. That's why unwrap functions throw an exception if the value is null: there is no other way to handle that case in a typed manner (well, I guess you also could loop forever...). In a nullable-by-default language? It is always fine to pass null. And it will not necessarily fail if a null is passed that shouldn't have been passed: it will only error when the reference is (tried to be) dereferenced.
That is, a type system is not the only tool we have in static analysis. So, I'm not completely convinced that moving to a type system is the best way to go.
Granted, it large part, it is clear it doesn't matter "what I think." Statically typed languages have a massive amount of support.
Specifically, majority of the static typing regimes seem to demand a change in how software is written. Whereas most of the static analysis seem to simply provide stronger guarantees on what you are already writing. That is a huge difference.
It's not the only tool but it's one of the better ones. It's faster, more reliable and more well-researched than almost any other tool.
What changed is that you are unable to propagate a null-that-shouldn't-be-null value to the rest of the program.
If you have a value V that might be null, and you are supposed to return that value itself (not wrapped in an Option or anything like that, ie a 'nullable type'), all you can really do is: actually return a value of type V, or throw an exception. That's it! (To throw an exception is also something that can be done at any time in most statically typed languages, for any reason whatsoever. Throwing exceptions is usually not a question of the type correctness of a program (maybe it is in the case of checked exceptions?).) You can not give null as an output: you are supposed to give V as the output, not a nullable V. So you know exactly where in your program you are doing something 'unsafely', i.e. where you are not rigorously handling every case of the value of some type. Though 'unsafe' is a bit misleading, since you may very well intend to just throw an exception in the (perhaps deliberately possible) case that a value is null.
Consider, in the Ariane Rocket incident, they were
using a dependently typed language. Didn't magically
prevent errors in assembly.
Since when is Ada a dependently typed language?The actual code that caused the overflow can be read here[^1] and the report on the overflow can be read here[^2].
[^1]: http://en.wikipedia.org/wiki/Cluster_%28spacecraft%29#Arithm...
[^2]: http://sunnyday.mit.edu/nasa-class/Ariane5-report.html
I can only guess that I got confused over reading about Agda at some point. Again, apologies.