Comment by wahern | Hacker News Reader

Comment by wahern | Hacker News Reader

The fact that Unicode codepoints were being passed to isspace instead of iswspace indicates the relevant code was already fubar'd.

> For example, isspace(0x01fe) is true. I can't figure out why this might be considered a whitespace character

Because the only valid values (independent of locale) that can be passed to isspace are 0 to UCHAR_MAX and -1/EOF, where UCHAR_MAX refers to unsigned char (usually 255), not Unicode character. Most implementations I've seen (glibc, musl, OpenBSD) index the passed value into a locale-specific array of length UCHAR_MAX + 1, possibly masking the index and/or return values. But TIL macOS (and possibly FreeBSD and NetBSD at some point, if not currently) had vestigial support for passing higher values as part of a presumably long-abandoned approach to I18N.

EDIT: FWIW, based on the glibc code (ctype/isctype.c),

    int
  __isctype (int ch, int mask)
  {
    return (((uint16_t *) _NL_CURRENT (LC_CTYPE, _NL_CTYPE_CLASS) + 128)
            [(int) (ch)] & mask);
  }

where isspace(c) seems to be translated to __isctype(c, _ISspace) there's a good chance the array is being overflowed. Without looking further (glibc isn't the easiest code to grok), I'd guess the array size is probably 128 + UCHAR_MAX with the offset of 128 (instead of 1) to handle the common case, especially on systems where char is signed, of people passing in negative values, though that only works for a locale like ASCII where -1/EOF and 255/(unsigned char)-1 aren't ambiguous.

FreeBSD layers narrow and wide character typing on top of a single common mechanism based upon 32-bit signed "runes". Basically, the 256 narrow characters are treated as the first 256 characters in the Unicode BMP, and an accident of implementation allows one to pass in other Unicode code points to the narrow character functions, given that the int and wint_t types are designed to be trivially convertible to a "rune".

Seems like instead of using boolean result better API should return tri-state: space, no space, bad request

So... True, False, and File Not Found?

Reference: https://thedailywtf.com/articles/What_Is_Truth_0x3f_

Reminds me of a joke in a John Meaney novel, a physicist discovers faster-than-light travel through a dimension with unusual properties, gaining inspiration from Java's booleans having three states (true, false and NullPointerException).

Hate to point it out, but java booleans can only be true or false. Booleans (capital B) can be true, false or null.

Method that is declared as only ever returning bool can also raise any exception that is an instance of java.lang.RuntimeException (which includes NullPointerException) this somewhat bizzare design which combines unchecked and checked exceptions is a source of major discussions since at least late 90's and has much to do with the current concept of null-safety.

final Boolean foo = True

…

if (foo.booleanValue()) { }

Can also NPE.

It can also raise Errors. I know because this one really poorly designed library I used would raise Errors when asking it to decode certain invalid files.

Yeah, but using a raw boolean or Boolean is not considered good practice. They’re up to at least four states now:

null, None, Some(TRUE) and Some(FALSE)

Of course, if you want more than that, you can have unboundedly many by invoking new Boolean(), which guarantees its return value is unique according to ==

This is already the case; "bad request" is usually returned as SIGSEGV / 0xC0000005 and similar.

No. Bad request here would be a recoverable, handleable error that doesn't tank the process. You are describing a failure mode for a bad pointer dereference, which generally should not be handled in-process and should take down the program.

It is not reasonable for isspace(), a simple function call with no pointer arguments, to fault like that, absent unlikely events such as a hardware issue or a bad stack pointer.

> It is not reasonable for isspace(), a simple function call with no pointer arguments, to fault like that

Well, if it internally uses a lookup table and doesn't do bounds checking, then it can fail this way, and that failure will propagate to the caller.

> You are describing a failure mode for a bad pointer dereference, which generally should not be handled in-process and should take down the program.

Maybe this is where we went wrong? Bad pointer access isn't fundamentally different from doing a bounds check and throwing an exception. Either of them will tank the process if not handled up the stack. But when you do a bounds check solely to prevent a bad pointer access crash, you're effectively re-doing the work the OS already does for you, paying a cost at runtime just to switch to a slightly different semantics. Maybe instead of doing that, we should have the platform provide better granularity of protection (as to e.g. avoid bad pointer access targeted to overwrite some other memory), and lean on it, instead of bolting extra layers of the same thing, just Invented Here on top?

You can't exactly lean on the OS to detect bad pointers references because it will only work some of the time.

The OS doesn't know where your array end, so the error would have to depend on whether the access is outside the memory allocated to the process. A function specification of "if the input is outside valid range, either return garbage or raise an exception" provides no value over "either return garbage or crash the program", because there is still the possibility of returning garbage.

> A function specification of "if the input is outside valid range, either return garbage or raise an exception" provides no value over "either return garbage or crash the program", because there is still the possibility of returning garbage.

Right. Yes, I was being a little tongue-in-cheek with the original comment, but only a little - imagine if there was a way for the OS / underlying runtime to prevent returning garbage (possibly by eagerly raising an exception). In that case, a lot of our error handling would be duplicating the work done by the OS. Now, with the garbage result being a possibility, we're still duplicating some of the work - we just can't really avoid it, in a kind of "50% of checks are redundant, we just don't know which ones" way.

I'm raising this as something to think about, that wasn't obvious to me until recently. I grew up dreading SIGSEGV and 0xC0000005. But recently, having no other choice but to trap some of those and similar exceptions (due to bugs in some proprietary third-party dependencies), I finally realized those are just error handling mechanisms, conceptually not any different from regular exceptions or Result<T, E> types. They're just implemented one layer below.

I don't think there is work duplication here, not in any meaningful way at least (and even in non meaningful ways I think you're also way off with your 50% figure - considering a 4kB page size and 64-bit pointer alignment, 0.2% would be more realistic).

SIGSEGV is a safety/isolation feature. It is a byproduct of physics limitations: if we had infinite RAM, we would allocate a separate 2^64 address space to each process and SIGSEGV would not exist. There are also zero guarantees that you will ever get a SIGSEGV, even if you do horrible, no-good, very bad things: that is entirely dependent on the OS and hardware. This allows for efficient implementations because the OS/hardware can implement checks at whatever granularity they deem appropriate. On some OS/hardware combinations you may never get a SIGSEGV at all!

On the other hand, ensuring errors on, amongst others, out-of-bounds array accesses, is much more expensive to do dynamically, and the OS can't really do it more efficiently than the compiler. In fact, it is the opposite: the compiler knows the specifics of the language and can exploit them to elide many boundary checks that the OS never could.

> Either of them will tank the process if not handled up the stack.

No. I think your mind must have its grooves set by high level languages to think this way. I was very deliberate when I said it's unwise to recover from bad pointer dereference in your own process. It is not safe and invites a world of pain. It isn't "an exception" that you can "catch and move on from". It is death. Yes you can install a handler for SIGSEGV or Win32 access violation. That doesn't make it a good idea.

You can handle it safely, but only from the safety of another process. Or the kernel can handle it in a page fault interrupt for a user process. This is because your own address space is known good in those cases. But handling it in your own address space, I would not advise that.

Too late to edit the post, but i wanted to add something i have done in a SIGSEGV handler in production, which is log a stack trace without allocating any memory, then exit. Anything beyond that is pushing it.

Yeah, unless you have a very specific architecture and compiler, etc., in mind. "Handling" SIGSEGV is madness -- you're usually already deep in UB territory if a SIGSEGV happens.

Re: Your use case, I believe the most recent C++ standard has a portable way to do stack traces, so yay!

> Well, if it internally uses a lookup table and doesn't do bounds checking, then it can fail this way, and that failure will propagate to the caller.

True, but I would say that wouldn't count as a reasonable implementation.

(I might be being overly nitpicky.)

> Well, if it internally uses a lookup table and doesn't do bounds checking, then it can fail this way, and that failure will propagate to the caller.

We're talking UB here, AFAICT. So it might do literally anything, including propagating to the caller. Or not. Or doing something entirely different, like returning true (or false)... or deleting your files.

One wishes that out-of-bounds access would guarantee a SIGSEGV, but here we are...

> Bad pointer access isn't fundamentally different from doing a bounds check and throwing an exception.

Performance is the difference. If you're doing bounds checks for every access you're going to be slower than a compiler which can prove (for a loop, say) that no OOB access can possibly occur during said loop and just remove the bounds checks.

> Either of them will tank the process if not handled up the stack.

Again, not true -- an exception has defined behavior, a stray pointer doesn't. (It could, conceivably, but it fundamentally doesn't in C.)

It is absolutely reasonable for function invocations that violate the specification to result in program termination. Attempting to handle and recover from the error at runtime is a) a waste of resources and b) prevents future expansion of the function that makes the values legal.

Reply on news.ycombinator.com