Hacker News Reader

If you don't have a null, then you have to set aside some value for the type as the default value. With null, you get a seg fault if you try and use it.

Without a null, you'll need to create a special default value which performs the same function - giving an error if you try to use it.

Floating point values have NaN for this, UTF-8 code units have 0xFF. The D programming language uses NaN and 0xFF to default initialize these types. NaN is good because it is "sticky", meaning if a result is computed that depended on a NaN, the result is NaN as well.

Some people complain about this, wanting floats to be 0.0 default initialized. But then it would be nearly impossible to tell if an unintended 0.0 crept into the calculation.

> More like "String and null"

Null's replacement, Maybe and Optional, still have the extra check.

Null certainly must exist at the target level. I'm not denying that. But languages can be designed to abstract nullability into controlled forms like Option/Maybe.

> Null certainly must exist at the target level.

Why? If anything, null must not exist at the target level- having a dedicated region of memory just for "accesses here are bad" at 0x0 isn't ubiquitous, and even having that at all isn't guaranteed (especially in embedded, where you might not get memory protection!).

Modern operating systems have to keep userland from mapping the 0x0 page because it turns out programmers just cannot keep themselves from dereferencing NULL, and that's often exploitable when done from the kernel.

>especially in embedded, where you might not get memory protection!

Or where there's perfectly valid memory mapped to address 0

The 8086 put the DOS vector table at address 0, in retrospect a truly terrible choice. Any null pointer writes would trash the operating system.

A much better design would have been to put the system ROMs at 0, where at least DOS would survive a null pointer.

Another way to address that problem is to not default initialise at all. There's then no need to worry about a default value for pointers, or about NaN vs. 0.0 for floats.

In addition, null doesn't always give a segfault, as I'm sure you're aware. In C and C++, using (dereferencing) null is undefined behaviour, and a segfault is the best-case result. The code may have been optimised so that there's not a direct dereference (which would segfault) but instead other operations that rely on the pointer being non-null.

Not to mention "fun" cases like this (contrived example) code-

  // Read sparse array from file

  uint64_t len = read_u64();
  uint8_t *buf = malloc(len);
  while (more_data_remaining_in_input_file()) {
    uint64_t pos = read_u64();
    if (pos >= len) { abort(); /* hackers detected! */ }
    buf[pos] = read_u8();
  }

which will very reliably compile into a write-what-where primitive when passed a gigantic `len`. malloc() fails and returns NULL (yes, it'll do this even on Linux when virtual address space is exhausted), and nasal demons emerge rapidly from there.

While this can happen, it's theoretical. In 40 years of dealing with null pointers, I've never seen one happen where the offset is outside the protected null page.

The reason is simple - very few allocated objects are that large, and for the arrays that are, they get filled from offset 0 forwards.

The real problem with C is not null pointers, it's buffer overflows caused by C arrays inescapably decaying to pointers, so array overflow checks can't be done:

https://www.digitalmars.com/articles/b44.html

Out of curiosity, have you been trying to make things work, or have you been trying to break things?

Because- here's the thing. I've been messing around with software security stuff for 5 years or so, and I've seen exploitable bugs related to a pointer unexpectedly being null twice.

There's a big difference between the kind of bugs you find "organically", when somebody's trying to use the software normally, and the kind of bugs you find when you're crafting (or fuzzing) absurd inputs that make no sense and that no ordinary software would produce. Perhaps this is why I've seen more of these bugs despite my much shorter career?

> I've seen exploitable bugs related to a pointer unexpectedly being null twice.

I've never heard of one, what I hear about endlessly are buffer overflows.

Can you give more information about these? I want to learn more.

Yeah, definitely! The one that comes to mind was in a game's texture loader - because of players being able to use custom "spray paint" images, this was exposed to untrusted code.

It unpacked mipmaps from a packed texture file into a malloc'd buffer, with an option to skip the first N mipmaps. (If I remember correctly, it'd then go back and upscale the largest mipmap to fill in the ones it skipped.)

Mipmaps were stored largest first, consecutively- so the first, say, 512x512xsizeof(pixel) would be the biggest mipmap, then you'd have 256x256xsizeof(pixel) bytes for the second-biggest one, etc, down to some reasonable (i.e. not 1x1px) minimum size.

The issue came when a texture's dimensions were specified as being so large that malloc'd fail and return NULL. Normally, this wouldn't be an issue (besides a denial of service) - but by skipping the first N mipmaps, you'd instead write to (where x and y are the dimensions of the texture)

  def addr(n, x, y, pixel_size_in_bytes=3):
    out = 0
    for i in range(n):
      out += x*y*pixel_size_in_bytes
      x >>= 2
      y >>= 2
    return out

By choosing x, y, and N carefully (I used a SMT solver to help) you could overwrite a function pointer and get it called before the later upscaling operation ran (since that would access 0x0 and crash).

It's definitely a unique bug, but this sort of thing does happen in real code.

Making malloc() and friends panic or similar on failure instead of returning NULL would fix most of these bugs- but it does sort of seem like the whole idea of sentinel values and in-band signalling is hazardous.

Go-style 'f, err := os.Open("filename.ext")' has its appeal from that perspective- you can forget to check "err" before doing things with "f", but I assume the Go ecosystem has good tooling to catch that.

Also probably worth noting that arguably this bug is related to C arrays being just pointers wearing fancy pants- as long as you can't get a slice where the pointer is NULL but the length is nonzero.

Yup, exploitable all right. Thanks for the explanation!

Usually, what I do is just make a shell around malloc() that checks for failure and aborts the program with a message. Then, and only then, it gets converted to a slice.

I'll still maintain, however, that buffer overflows are a couple orders of magnitude more common. You case involves loading a file essentially backwards, which seems awful obscure. When I load files piecemeal, I'm using memory mapped files, not a malloc'd buffer.

What? I don't get why you're saying "you have to". Besides, with null you don't necessarily get a segfault, in C you get undefined behavior

Rust skips nulls & statically disallows use of variables before initialization

> Rust skips nulls & statically disallows use of variables before initialization

That doesn't mean you won't need a "not a valid object" for a type. That's why there all these Optional and Maybe constructions - you're still checking for it.

On the other hand, Rust doesn't let you use an `Option<T>` as if it was a `T`, so you can't just forget to check. And most of your code won't be using Option, it's only used right where it's needed.

> you're still checking for it

Where it makes sense to because you actively want it, nowhere else, and you can't accidentally forget it without making code smells bad enough to set off the fire alarm.

In my experience, Option/Optional/Maybe of a type T comes up a lot less than the type T itself. That is, needing "not a valid object" is a rarer than only requiring objects that are always valid.

Those construction exactly exist so that you have to check for it. They force the programmer to think about the "maybe invalid" state, whereas built-in-null doesn't prompt this (and, indeed, since might-be-null is the rare state, people often just assume they're working with a valid instance of the type and don't handle null at all).

> That doesn't mean you won't need a "not a valid object" for a type.

This confuses two different concepts, and that confusion is precisely the problem with "null".

Most types neither need, nor have, a "not a valid object" value. The "not a valid object" value is typically disjoint from the type.

Null can't be used instead of some other type, as though it were that type. Instead, you have to perform a comparison to decide whether you're dealing with null, or a value of the expected type.

The problem with null, in languages that have it, is the inability to statically prove that terms cannot be null.

This means null is a pervasive possibility that almost always has to be checked for, which is an unnecessary source of bugs. Strictly speaking, you always have to check for null, except in those cases where you can prove (informally, since the compiler doesn't help with this) that you already did that.

> That's why there all these Optional and Maybe constructions - you're still checking for it.

The point is that option types allow you to choose whether you want to allow non-values at a particular term. If you don't - e.g. if a function returns a value not wrapped in an option type - then you're statically guaranteed to be able to use that value without having to check for null.

The only place where you need option types are where you're explicitly allowing for an optional value. And in that case, the option type helps because it statically disallows you from treating the value as though it were an instance of the underlying type.

So while it might seem like "you're still checking for it," there's an important difference in the context of that check - the compiler will give you an error if you try to use the value without unwrapping the option.

Equating Option<T> to T is like equating T[] to T, so you're saying everything should be arrays (& in fact, C goes that far by making it unclear whether int* is a reference to one int or many ints)

> Null's replacement, Maybe and Optional, still have the extra check.

That's a straw man. You would also have to check for Null / NaN / 0xFF to see if you're dealing with a valid object.

That's true. Though for Null and NaN, the hardware does that for you.

Sometimes you do need a marker for "not initialized", but the problem is languages where every type includes this "not initialized" value. This would be like a language were every value should be a Maybe (except that would be an infinite regress).

For all variables which are definitely assigned (e.g. "int x = 17") you don't need a special "uninitialized" value, and the type system should be able to reflect that.

I guess floating point numbers are a special case, since the standard mandates that it includes the NaN value.

> I guess floating point numbers are a special case, since the standard mandates that it includes the NaN value.

NaN is more useful than that. Consider if you've got a sensor array. The data from the array is crunched to produce a result. Suppose there are a few failed sensors in the array. By using NaN's for those sensors' data, you can then tell what part of your crunched output is dependent on that bad data.

If you'd just used 0.0 for the bad sensor data, you'd have no idea.

Oh I agree about that, and I agree that NaN is a good default for an uninitialized floating point variable.

My point was that you don't always, with every variable, need to be able to express an "undefined" or "invalid" value. The type system should be able to express if a given value is nullable or not.

> The moment you pass that value out of the context of the function, you need to remember that null means EndOfString

Good point, and I don't disagree, but it's also interesting to note that once you return 4 from CountChildren(), you similarly have to remember that it represents the number of children.

Except you can always assign the result of CountChildren to a `childrenCount` variable. Is there a variable name you could give to the result of `parse` that'd accurately describe the error states?

I don't see a valid use case for null. It's just a easy-to-make billion dollar mistake.

Null exist because people are being lazy, and don't want to think reference type T to be T | null. Instead, in most of the cases they ignore null unless they know it will crash.

It seems to be every use case for null could be handled well in Haskell, which means null is not mandatory in language design. Everywhere using null can be replaced with Maybe or concrete union type like Int | NaN just fine.

No. The problem is conflating "logical null" (the absence of data) with "technical null" (a pointer to nothing).

Logical null is often essential for expressing business logic in code. Technical null is (as far as I can tell) a C implementation detail that probably never should exist in a higher-level language.

The Haskell equivalent of logical null is Nothing. There is no technical null because there is no need to represent that one specific implementation detail in Haskell.

As for using NaN to represent logical null instead of a special reserved value, that has a lot of ergonomic problems. Anyone who has used Python for data analysis can attest.

I think we need more nulls then. In a sense. Consider a ‘cell’ (value box).

1. Null for no value there. But it can be set, that’s our conventional null.

2. Null for many values from a filter that cannot fit into a single cell. Can be set all at once. Not very null, but cell’s situation-wise it is.

3. Null for no value that cannot be set, e.g. filter produced no values to show or edit at all.

4. Null that is not equal to itself for relational joining in non-sql. (3=4? May be to some extent)

5. Null for a value that must be there, but is not yet (Promising Null). Cannot be set and throws on any comparison, since semantics are unclear until resolving. (4,5) require special isoftype() tests to check their type.

5a. Null for a value that is Promised only on access. Doesn’t throw, but instead yields until it’s done (or throws on failure). You can still check if it is already here, e.g. to choose a better access pattern.

6. Null for a missing key, as opposed to a key holding any of null values above. Absense of a cell itself. Not really 6, but another dimension (remember Perl exists $hash{key}, Js, Python ‘key’ in object, etc, that is hard to test from the object itself).

I’m not a type scientist, but it is clear that semantics of a value in reality decompose in much more ways than just null/non-null. Objects (including primitives) should have at least so much null values when used not in a vacuum, but in loading cross-referenced dataset aggregations that appear in our programs everywhere. I’m not really making any counter-argument here, just my daily job observations to discuss!

But I believe that a language that could cook it all right at the core level and remain convenient to program in will be a million years’ win.

Which is why variants/discriminated unions are so powerful. A common "hump", if you will, that beginners get over with OCaml is an over reliance on booleans. A beginner will write a function that takes an isOpen argument. A more advanced programmer will model the state as a discriminated union that is either Open or Closed.

Combine that with generics and you can get generic unions like Maybe which allow you to create sets of states that can be composed with any type.

Also from Ward Cunningham is the CHECKS Pattern Language (published as far back as 1994), which developed novel ideas about value objects well beyond the mainstream of the day. The examples are in Smalltalk but the uniform abstraction level for values is an instantly applicable concept, allowing generic treatment of value objects, and is probably more accessible to modern audiences comfortable with option types and so on than it was at the time.

http://c2.com/ppr/checks.html