Hacker News Reader

NULL is just a tool. Non-optional nullability on all pointer types is a blight. 99% of APIs want to say "returns a guaranteed-valid pointer", and "takes a guaranteed-valid pointer", and have the language enforce those guarantees. C and its siblings just give you a "might-be-valid pointer" type, which introduces NULL into every API you create, whether you want it there or not.

I do agree that there's no fundamental difference between using a "might-be-valid pointer" vs. a sum-type (Option/Maybe/etc.) to indicate nullability. However, it's usually easier to get static analysis of the sum type—most type-inferencing compilers can ensure that when you destructure a sum type, you provide match-clauses for every possible instantiation of the type (i.e., for an Option type, they ensure you do the equivalent of a NULL check, and handle both the NULL and non-NULL cases), and so forth.

What I'd really like to see, though, is a language with data-parameterized "types" that unfold (as a last resort, when the concrete type cannot be inferred at compile-time) into runtime data-comparison branches. You should be able to have a raw may-be-valid address with no metadata tag, and then treat that data as having a separate type (a Just or a Nothing) depending on whether the memory address itself is nonzero. You write match clauses; the compiler turns those into NULL checks.

(As an aside, such a language would also allow for things like a type for "Float that only has an integer component", or "Even number", or "Positive number", or "Number in the range 5..23", or "String of length 5", or "String that encodes a valid Unicode codepoint sequence", etc. The type system would become your swiss-army knife for expressing invariants.)

> What I'd really like to see, though, is a language with parameterized typing with hybrid static+runtime enforcement. You should be able to be holding a raw address in memory, and then have that become one of two strong types at the language level depending on the whether that memory address is nonzero—"lifting" the NULL check into the type system, rather than replacing the NULL check with extra type annotation.

In Rust, the compiler optimizes pointer types wrapped in Options to nullable pointers. Is this the kind of thing you meant?

Possibly—does the Rust FFI allow you to import a C library, writing its foreign-function declarations such that the nullable pointers it returns are transformed into, from Rust's perspective, Option-typed non-nullable pointers, with no runtime memory overhead? And then, if so, does Rust further guarantee that those "virtual" Option-typed pointers can be passed back to the same API "just the way it sent them"?

That's sort of what I mean, here: the nullable ptr as the "canonical, machine-level" representation, with the type serving as a language-level abstraction, effectively a "lie" because it has no real effect on the type system, and only serves as syntax sugar for generating runtime checks.

The Rust FFI takes all pointers as "raw pointers". The reason for this is that Rust's various safe pointer types have a lot of extra conditions on them to enforce memory/data race safety without garbage collection. For example, some pointers are uniquely owned and will be freed when they go out of scope, some can't be stored because they are "borrowed", some can't be mutated because they're shared etc. When Rust gets a pointer from another language like C (that obviously doesn't have such semantics), whoever is writing the FFI has to decide what category these raw pointers fall under before passing them back to "safe" code.

Depending on the type of semantics you are attaching to the pointer, you can use something like this[1] function, which does exactly what you're asking. It is a runtime no-op, but it coerces the unsafe pointer type so that memory safe code can use it (based on your word that it can be interpreted in this way).

[1]: https://doc.rust-lang.org/std/primitive.pointer.html#method....

To be clear, I wasn't so much imagining using the FFI declaration to vouchsafe the opaque handle into a completely Rust-nativized memory-safe owned reference; but rather, as taking the opaque nullable pointer from the FFI, and unwrapping it exactly one level into an Option<opaque non-nullable pointer>—that is, turning a nullable FFI "handle" into an Option<non-nullable FFI "handle">.

Such an FFI function declaration really can't require that the thing returned inside the Option be a dereference of the pointer: in the happy case, it's still an opaque FFI handle, so dereferencing it just exposes an opaque struct you shouldn't touch; and in the unhappy case, it's still an invalid pointer, just one that's invalid because somebody else (like the library that gave it to you) freed it. The raw pointer itself, even if non-NULL, is still "tainted"; you still want to apply manual runtime checks to it before casting it to a Rust reference. (And, past that, in both cases you still probably want to deal with the pointer mostly just by passing it back to other FFI functions from the same library, treating the pointer itself as an identifier, a key the API associates with something.)

I'm emphasizing this distinction because the NULLness of the pointer is, from what I can see, "outside" the lifetime of the reference, and even outside the "identifier-ness" of the handle. Knowing the validity of the pointer requires knowing about ownership/lifetime; the pointer might point to memory that has been free()ed. But you can know that the pointer is intentionally NULL (and therefore not a valid handle/identifier from the FFI's perspective) without attempting to construct a reference.

Effectively, a [nullable] raw pointer is the moral equivalent of a tagged data structure. There's a piece of metadata you can get from the tag: when it's zero, the contained raw pointer is intentionally invalid; when it's nonzero, the contained raw pointer is only maybe-invalid. You can destructure and act on the "tag" without doing anything else to the [non-nullable] raw pointer "stored inside".

I wouldn't expect most languages to care about a distinction this fine, because if a language has references, it really only wants to think about "pointers: maybe invalid" and "references: always valid". But Rust is in the unique position of trying to work "directly with C" without the impedance mismatch of glue like SWIG. And C provides many libraries that return nominally opaque handles, but expect their consumers to notice if those handles are NULL (which happens often, when e.g. a factory-function from such a library is called with NULL arguments, or unexpectedly receives a NULL from a library it itself consumes.)

Well, the answer to that is somewhat complicated. In Rust, a raw pointer has the same semantics of a C pointer. Could be null, could point to nowhere, the object pointed to could live temporarily, could live forever etc. However, there aren't any pointers which only weaken some of these conditions.

The function I linked takes a possibly null raw pointer, and vouches that:

1. It points to null or something. 2. The object it points to (if it is not null) will live at least as long as the current scope. 3. The object it points to will not be mutated as long as the current scope is active.

Practically speaking, as long as you don't do anything with the resulting Option<&T> other than check if it is empty, you're safe. In addition, Rust will preserve the internal pointer address, so you could bring it back to the FFI. So you could treat it consistently as such, you'll be fine. I'm not sure if the language spec guarantees that, because that's not how Rust references are designed to be used, but it won't matter in practice.

For what you want, you could just create a generic structure that holds a pointer to the given type, and lets you ask if the contained pointer is null. That way you won't add any false guarantees to the pointer (e.g. immutability, lifetime, etc.).

The problem is NULL does not "fail fast". You could get a NULL pointer as an argument, and then not use that value until much later. Now when the bug shows up, you'll have to work back through your code to even figure out where it came from.

> Use of a class type implementing an interface as a "null type" is a very good strategy.

Have you ever programmed Objective-C? It uses the null object pattern (called "nil") for all Objective-C objects. This makes it really easy to put big logic errors into your code, since calling upon a nil will just give you more nil. So now you have to wait even longer until a fatal error pops up, and then find where on earth the original nil came from.

> NULL and nullptr are convenient ways to do that without a ton of extra code.

How do you figure? If malloc can return NULL and you want a sane error message, you need to write a NULL check whenever you do a malloc. Otherwise, your user will just get a segfault, instead of a notification that it was memory they ran out of. So why not do that by default, and for the few places where you have prepared yourself to handle out of memory conditions use a different malloc which returns a value that must be explicitly checked?

Because I don't have a system library that provides a different malloc. It's a property of my API that malloc returns NULL and I don't have any control over that. What I do have control over is what I do with data I get out of malloc. I have to check, somehow, for abnormal conditions. That is a requirement for using the malloc API. I get that this makes it hard to debug in some cases, so I might avoid that by religiously checking for NULL return from malloc and crashing immediately with a sensible error message. In the cases where getting a NULL return is not an error, like if I wrote a program to use as much heap as it can, I would ignore the NULL and proceed with the resources I can actually get.

What stops you from writing an API on top of the system malloc that has two malloc functions like I described above, with return types that have unambiguous nullity? Then your religious NULL checks won't be duplicated (and you won't be able to stray from the righteous path and forget any).

An optional type's "none" value represents the same properties that NULL does. The difference is that the optional type better expresses what's going on.

NULL breaks the type system. Now every pointer type is actually two types: the normal type, and the NULL type. You can dereference or call methods on one, but not the other, and only a runtime check will tell you what you're dealing with. Any time you see:

    Object *

You cannot read that as "pointer to Object." You have to read it as "pointer to Object, or NULL."

This causes three big problems. One is that it's way too easy to forget to check for NULL. The second, related problem is that without a way to express a non-nullable pointer type, you have to rely on documentation and convention to express the difference. The third is that since only pointer types are nullable, you have to reinvent the wheel any time you you want to express the concept of "null" for a non-pointer type.

To illustrate, you point out malloc which returns NULL when it runs out of address space. What's the appropriate return type?

    void *

Now let's say you had some other memory allocation call which could never return failure to the caller. Maybe it aborts execution on failure, or maybe it retries continually until memory becomes available. What's the appropriate return type?

   void *

Oops. Same problem with parameters. Here's a function:

   void f(void *ptr)

Are you allowed to pass NULL? Who knows, go RTFM. If the answer is "no," what happens if you pass NULL anyway? The compiler certainly won't catch it, and who knows what will happen.

NULL is "just a tool" but it's not a very good tool. It breaks the type system in a fundamental and painful way.

Edit: whew, HN's markup makes it kind of hard to talk about C pointers!

Right, but in malloc's case, it needs to express something about the fundamental type of all software, which is memory. There is nothing more fundamental to the software system. When you run out of memory and fail to check for that condition, the only right and proper thing to do is crash. NULL is a handy way to enforce that fundamentally, in that dereferencing a null pointer guarantees a crash.

Part of programming is about memory management. You can hide it with a sophisticated compiler but it's always going to be there, and there are always going to be two types of memory: that which is available, and that which is not.

>NULL is a handy way to enforce that fundamentally, in that dereferencing a null pointer guarantees a crash

In the context of C and C++, dereferencing a null pointer does not guarantee a crash, or anything else for that matter. In fact, not only does it not give you any guarantees, it nullifies any guarantees you might otherwise have had, because the behaviour of a program that dereferences a null pointer is undefined. Now, an implementation is of course free to make guarantees about programs that have undefined behaviour, but none I know of does.

>Part of programming is about memory management. You can hide it with a sophisticated compiler but it's always going to be there, and there are always going to be two types of memory: that which is available, and that which is not.

This is not true. It is perfectly possible to write useful programs that never have to deal with unavailable memory or even memory management at all past compile time, even in C. This is quite common (as are implementations that don't crash programs which dereference null pointers) when working with embedded systems, for example.

Dereferencing NULL does not guarantee a crash, actually. And this isn't one of those theoretical things where people say "that's undefined behavior!" but real-world systems always do a certain thing. Non-crashing NULL dereferences are real events that cause real problems. Just toss "NULL dereference exploit" into your favorite search engine for a vast list of examples.

Here's another potential problem. Let's say you write a little bit of code to copy an array into some dynamic storage, basically strdup for non-strings:

    void *memdup(const void *source, size_t length) {
        void *destination = malloc(length);
        memcpy(destination, source, length);
        return destination;
    }

But then you're a little paranoid about a NULL dereference not crashing, and you want to call some special failure handler that will log more information about what happened, so you add this bit of code in the middle:

    if(destination == NULL) {
        log_failure();
        abort();
    }

Oops, you've written a bug! It's possible for malloc to return NULL when you haven't run out of memory, specifically when you pass 0. The FreeBSD man page calls this "a silly response to a silly question." Zero-length arrays are a perfectly valid concept, and there's no reason this memdup function should have trouble with them, but now it does. Maybe. Depending on your malloc implementation.

Using an optional type would do away with all of these problems. It would guarantee that dereferencing a null pointer would actually halt execution, rather than just "probably," and it would highly encourage distinguishing between "an error occurred" and "no error occurred, but you didn't actually ask for any memory."

Yes, the underlying NULL value will always be there. Any reasonable system will represent the "null" value of an optional pointer type using the underlying NULL value. But in terms of the type system, NULL is terrible and optionals are much better. There's no real downside to it, and you can rest easy knowing that you're still working with 0x0 in the end.

It doesn't break the type system, it's that type system is lacking! There's nothing wrong with the existence of NULL, the problem is, like you say, that in C (and many other languages) there is no ability in the type system to specify non-NULL pointer or reference types.

One is that it's way too easy to forget to check for NULL.

Null-checks only make sense when you expect a null value. If you don't, and you get one, then either your code should be expecting one and has to do some processing differently, or something else is broken.

If the answer is "no," what happens if you pass NULL anyway? The compiler certainly won't catch it, and who knows what will happen.

In all likelihood, a crash when it tries to do something with whatever the pointer should be pointing at, in which case you should be asking why your code is giving it a null.

And when do you expect a NULL value? If you use an optional type, then you expect a NULL value if and only if the type in question is an optional. If you forget the check, or if you check a value that can't be NULL, the compiler will tell you.

Beware programming errors which reveal themselves as "in all likelihood," because that hides an entire universe of difficult-to-debug behavior, and sometimes behavior that can be subverted.

Optionals solve these problems, and solve them far better than the "be careful not to do that" approach you describe here.

> NULL breaks the type system. Now every pointer type is actually two types: the normal type, and the NULL type. You can dereference or call methods on one, but not the other, and only a runtime check will tell you what you're dealing with.

But option types do the same thing. The option type is actually two types. You can use one, but no the other, and only a runtime check will tell you what you're dealing with.

It may be more convenient than dealing with NULL, but it's doing the same thing...

The difference is that in C and derived languages, nullability is completely tied to pointer types. All pointer types are nullable, and no non-pointer types are nullable. Having nullability/optionality be a separate concept makes everything nicely composable and solves some big problems.

If you have an interface which returns a pointer that's never NULL, you still have to return a nullable pointer. The non-nullability of the value can only be encoded in documentation or naming conventions. Because of this, the language has to make it easy to use pointers without first checking for NULL, otherwise all these interfaces would become a giant pain in the ass to use. And that means that when you're working with an interface which can return NULL, you have nothing that forces you to check for that. You just have to remember. It's similar for function parameters.

Optional types also work for non-pointer types, which C's NULL simply cannot do. That's a major plus, beyond the problem of all pointers being nullable.

You're right that an optional type is actually two types. But it's designed such that you can't get at the underlying runtime type without first checking to see what it is. That's a huge difference.

> All pointer types are nullable, and no non-pointer types are nullable.

It's also error numbers. Yeah, they're just an int as far as types go, but they also have nullable semantics, with 0 being "no error". So it's not just pointers (quite).

> You're right that an optional type is actually two types. But it's designed such that you can't get at the underlying runtime type without first checking to see what it is. That's a huge difference.

Agreed.

I disagree that error numbers are nullable. It's just an integer, and there's nothing special about the zero integer. It's not like pointers where if you dereference NULL, you've hit a special case where anything could happen. There's a difference between a language-level NULL, as pointers have, and a value with no special language meaning which you give special meaning at a higher level.

If you want an example of a non-pointer nullable type, float and double probably qualify. Both can be NAN, which is effectively NULL for floats. Like NULL, NAN behaves completely differently from any other value and only convention and documentation tell you whether any particular place might produce or accept it.

Don't forget about signalling vs. non-signalling NANs...

Dereferrencing NULL will not always cause your program to crash.

You have intrigued me, as what you say is true, but I wish you would elaborate. There are some architectures for which dereferencing an address of 0 does not cause an exception. If you are writing code for those architectures (or code that is meant to be portable), then you can't rely on dereferencing NULL to crash. This is universally well known amongst programmers who work on those architectures.

It's been more than a decade since I did any C/C++, so I wonder if you were referring so something else that I'm not aware of.

Generally it is not an "architecture" that makes this fail, it is a property of the operating system or even only the linker. If you look at a Mach-O executable on OS X, you will see a __PAGEZERO segment that explicitly causes the dynamic linker to put an invalid block of memory at address 0: for backwards compatibility reasons, in 32-bit processes, without that totally optional load command, and even with that command if you choose to override it at runtime using mmap or any number of mach memory management functions, you can allocate memory at that address. (Apple mitigates this in 64-bit processes by having the operating system disallow mapping memory to this location.) In fact, this is a technique commonly used in many kernel exploits on both Linux and OS X, and was even used just a few months ago in the "tpwn" exploit of OS X 10.10.5, prompting the researcher who disclosed that vulnerability to also release NULLGuard, whose entire purpose is to help protect your system by disallowing processes from executing if they are missing a __PAGEZERO segment.

https://github.com/kpwn/NULLGuard

It can also be the compiler. If it observes you dereferencing a pointer, it can elsewhere assume that pointer is not null. If that's not the case, things can get messy.

I agree it's possible to design a compiler that doesn't crash on a NULL pointer and still call the language C. Or certain optimizations could trigger that confuse a naive C programmer.

But modern hardware is frequently designed so that dereferencing a zero pointer causes a crash. Even embedded systems will often treat the first couple hundred bytes or so after 0 specially, since it catches so many bugs.

I think it's generally safe to assume you'll get a crash if you actually dereference a null pointer.

This is a false assumption that directly leads to exploitable software. See my reply to a sibling comment for more details, but I will add here (as you went in a slightly different direction) that on many embedded systems (such as ARM) the address 0 is not "special" in that it is disallowed but is "special" in that it stores particularly important control structures for the CPU, such as its interrupt vector table: one of the bootrom exploits used on the iPhone took advantage of this (though sadly I'm not remembering exactly which one, though the documentation for 24kpwn mentions this mapping). It isn't until you are running in a high-level operating system, and even sometimes not even then (such as current versions of OS X, which rely on cooperation between the dynamic linker and the compiler to implement this feature, as I describe in my linked reply), that you have some (limited...) safety from NULL pointer dereferences. (I say "limited", as sometimes people store egregiously large objects that are larger than a couple pages of memory and some dereferences are negative.)

https://news.ycombinator.com/item?id=10531319

https://www.theiphonewiki.com/wiki/0x24000_Segment_Overflow

Funny story: a customer porting from an OS that didn't leave NULL unmapped to an OS that did, asked us to tell them how to map the zero page, as their program had spurious NULL pointer accesses that they didn't want to fix.