Hacker News Reader

Comment by WalterBright | Hacker News Reader

If you don't have a null, then you have to set aside some value for the type as the default value. With null, you get a seg fault if you try and use it.

Without a null, you'll need to create a special default value which performs the same function - giving an error if you try to use it.

Floating point values have NaN for this, UTF-8 code units have 0xFF. The D programming language uses NaN and 0xFF to default initialize these types. NaN is good because it is "sticky", meaning if a result is computed that depended on a NaN, the result is NaN as well.

Some people complain about this, wanting floats to be 0.0 default initialized. But then it would be nearly impossible to tell if an unintended 0.0 crept into the calculation.

> More like "String and null"

Null's replacement, Maybe and Optional, still have the extra check.

View on HN

Another way to address that problem is to not default initialise at all. There's then no need to worry about a default value for pointers, or about NaN vs. 0.0 for floats.

In addition, null doesn't always give a segfault, as I'm sure you're aware. In C and C++, using (dereferencing) null is undefined behaviour, and a segfault is the best-case result. The code may have been optimised so that there's not a direct dereference (which would segfault) but instead other operations that rely on the pointer being non-null.

Not to mention "fun" cases like this (contrived example) code-

  // Read sparse array from file

  uint64_t len = read_u64();
  uint8_t *buf = malloc(len);
  while (more_data_remaining_in_input_file()) {
    uint64_t pos = read_u64();
    if (pos >= len) { abort(); /* hackers detected! */ }
    buf[pos] = read_u8();
  }

which will very reliably compile into a write-what-where primitive when passed a gigantic `len`. malloc() fails and returns NULL (yes, it'll do this even on Linux when virtual address space is exhausted), and nasal demons emerge rapidly from there.

While this can happen, it's theoretical. In 40 years of dealing with null pointers, I've never seen one happen where the offset is outside the protected null page.

The reason is simple - very few allocated objects are that large, and for the arrays that are, they get filled from offset 0 forwards.

The real problem with C is not null pointers, it's buffer overflows caused by C arrays inescapably decaying to pointers, so array overflow checks can't be done:

https://www.digitalmars.com/articles/b44.html

Out of curiosity, have you been trying to make things work, or have you been trying to break things?

Because- here's the thing. I've been messing around with software security stuff for 5 years or so, and I've seen exploitable bugs related to a pointer unexpectedly being null twice.

There's a big difference between the kind of bugs you find "organically", when somebody's trying to use the software normally, and the kind of bugs you find when you're crafting (or fuzzing) absurd inputs that make no sense and that no ordinary software would produce. Perhaps this is why I've seen more of these bugs despite my much shorter career?

> I've seen exploitable bugs related to a pointer unexpectedly being null twice.

I've never heard of one, what I hear about endlessly are buffer overflows.

Can you give more information about these? I want to learn more.

Yeah, definitely! The one that comes to mind was in a game's texture loader - because of players being able to use custom "spray paint" images, this was exposed to untrusted code.

It unpacked mipmaps from a packed texture file into a malloc'd buffer, with an option to skip the first N mipmaps. (If I remember correctly, it'd then go back and upscale the largest mipmap to fill in the ones it skipped.)

Mipmaps were stored largest first, consecutively- so the first, say, 512x512xsizeof(pixel) would be the biggest mipmap, then you'd have 256x256xsizeof(pixel) bytes for the second-biggest one, etc, down to some reasonable (i.e. not 1x1px) minimum size.

The issue came when a texture's dimensions were specified as being so large that malloc'd fail and return NULL. Normally, this wouldn't be an issue (besides a denial of service) - but by skipping the first N mipmaps, you'd instead write to (where x and y are the dimensions of the texture)

  def addr(n, x, y, pixel_size_in_bytes=3):
    out = 0
    for i in range(n):
      out += x*y*pixel_size_in_bytes
      x >>= 2
      y >>= 2
    return out

By choosing x, y, and N carefully (I used a SMT solver to help) you could overwrite a function pointer and get it called before the later upscaling operation ran (since that would access 0x0 and crash).

It's definitely a unique bug, but this sort of thing does happen in real code.

Making malloc() and friends panic or similar on failure instead of returning NULL would fix most of these bugs- but it does sort of seem like the whole idea of sentinel values and in-band signalling is hazardous.

Go-style 'f, err := os.Open("filename.ext")' has its appeal from that perspective- you can forget to check "err" before doing things with "f", but I assume the Go ecosystem has good tooling to catch that.

Also probably worth noting that arguably this bug is related to C arrays being just pointers wearing fancy pants- as long as you can't get a slice where the pointer is NULL but the length is nonzero.

Yup, exploitable all right. Thanks for the explanation!

Usually, what I do is just make a shell around malloc() that checks for failure and aborts the program with a message. Then, and only then, it gets converted to a slice.

I'll still maintain, however, that buffer overflows are a couple orders of magnitude more common. You case involves loading a file essentially backwards, which seems awful obscure. When I load files piecemeal, I'm using memory mapped files, not a malloc'd buffer.

What? I don't get why you're saying "you have to". Besides, with null you don't necessarily get a segfault, in C you get undefined behavior

Rust skips nulls & statically disallows use of variables before initialization

> Rust skips nulls & statically disallows use of variables before initialization

That doesn't mean you won't need a "not a valid object" for a type. That's why there all these Optional and Maybe constructions - you're still checking for it.

On the other hand, Rust doesn't let you use an `Option<T>` as if it was a `T`, so you can't just forget to check. And most of your code won't be using Option, it's only used right where it's needed.

> you're still checking for it

Where it makes sense to because you actively want it, nowhere else, and you can't accidentally forget it without making code smells bad enough to set off the fire alarm.

In my experience, Option/Optional/Maybe of a type T comes up a lot less than the type T itself. That is, needing "not a valid object" is a rarer than only requiring objects that are always valid.

Those construction exactly exist so that you have to check for it. They force the programmer to think about the "maybe invalid" state, whereas built-in-null doesn't prompt this (and, indeed, since might-be-null is the rare state, people often just assume they're working with a valid instance of the type and don't handle null at all).

> That doesn't mean you won't need a "not a valid object" for a type.

This confuses two different concepts, and that confusion is precisely the problem with "null".

Most types neither need, nor have, a "not a valid object" value. The "not a valid object" value is typically disjoint from the type.

Null can't be used instead of some other type, as though it were that type. Instead, you have to perform a comparison to decide whether you're dealing with null, or a value of the expected type.

The problem with null, in languages that have it, is the inability to statically prove that terms cannot be null.

This means null is a pervasive possibility that almost always has to be checked for, which is an unnecessary source of bugs. Strictly speaking, you always have to check for null, except in those cases where you can prove (informally, since the compiler doesn't help with this) that you already did that.

> That's why there all these Optional and Maybe constructions - you're still checking for it.

The point is that option types allow you to choose whether you want to allow non-values at a particular term. If you don't - e.g. if a function returns a value not wrapped in an option type - then you're statically guaranteed to be able to use that value without having to check for null.

The only place where you need option types are where you're explicitly allowing for an optional value. And in that case, the option type helps because it statically disallows you from treating the value as though it were an instance of the underlying type.

So while it might seem like "you're still checking for it," there's an important difference in the context of that check - the compiler will give you an error if you try to use the value without unwrapping the option.

Equating Option<T> to T is like equating T[] to T, so you're saying everything should be arrays (& in fact, C goes that far by making it unclear whether int* is a reference to one int or many ints)

Null certainly must exist at the target level. I'm not denying that. But languages can be designed to abstract nullability into controlled forms like Option/Maybe.

> Null certainly must exist at the target level.

Why? If anything, null must not exist at the target level- having a dedicated region of memory just for "accesses here are bad" at 0x0 isn't ubiquitous, and even having that at all isn't guaranteed (especially in embedded, where you might not get memory protection!).

Modern operating systems have to keep userland from mapping the 0x0 page because it turns out programmers just cannot keep themselves from dereferencing NULL, and that's often exploitable when done from the kernel.

>especially in embedded, where you might not get memory protection!

Or where there's perfectly valid memory mapped to address 0

The 8086 put the DOS vector table at address 0, in retrospect a truly terrible choice. Any null pointer writes would trash the operating system.

A much better design would have been to put the system ROMs at 0, where at least DOS would survive a null pointer.

Sometimes you do need a marker for "not initialized", but the problem is languages where every type includes this "not initialized" value. This would be like a language were every value should be a Maybe (except that would be an infinite regress).

For all variables which are definitely assigned (e.g. "int x = 17") you don't need a special "uninitialized" value, and the type system should be able to reflect that.

I guess floating point numbers are a special case, since the standard mandates that it includes the NaN value.

> I guess floating point numbers are a special case, since the standard mandates that it includes the NaN value.

NaN is more useful than that. Consider if you've got a sensor array. The data from the array is crunched to produce a result. Suppose there are a few failed sensors in the array. By using NaN's for those sensors' data, you can then tell what part of your crunched output is dependent on that bad data.

If you'd just used 0.0 for the bad sensor data, you'd have no idea.

Oh I agree about that, and I agree that NaN is a good default for an uninitialized floating point variable.

My point was that you don't always, with every variable, need to be able to express an "undefined" or "invalid" value. The type system should be able to express if a given value is nullable or not.

> Null's replacement, Maybe and Optional, still have the extra check.

That's a straw man. You would also have to check for Null / NaN / 0xFF to see if you're dealing with a valid object.

That's true. Though for Null and NaN, the hardware does that for you.

Reply on news.ycombinator.com