Undefined behavior in C is a reading error

from ISO/IEC 9899:2011 "Programming Languages -- C"

    3.4.3

    1 undefined behavior behavior, upon use of a nonportable or erroneous program   construct or of erroneous data, for which this International Standard imposes no requirements

    2 NOTE Possible undefined behavior ranges from ignoring the situation completely with unpredictable results, to behaving during translation or program execution in a documented manner characteristic of the environment (with or without the issuance of a diagnostic message), to terminating a translation or execution (with the issuance of a diagnostic message).

It doesn't look like a leap to go from this definition to that of ignoring the situation completely with unpredictable results.

> and the commtitee

I'd argue that for a document like the C standard, if there's a well-known intended meaning, that is the meaning of the document - any other interpretation is purely academic.

> In particular, nothing in the wording of the standard expliclty says that an implementation is expected to assume UB doesn't happen, or that a standard-conforming program can't have UB.

An implementation isn't expected to assume that UB doesn't occur, but it is allowed to assume that.

With regard to programs, the C standard has two different notions of conformance (cf. chapter 4 Conformance). There are strictly conforming programs, which may not rely on anything that would depend on the specifics of a particular C implementation, which of course includes UB. Strictly conforming programs are thus guaranteed to have the same behavior on all conforming C implementations. Then there is the larger class of conforming programs, which is defined as programs acceptable to a (particular) conforming C implementation.

Strictly conforming C programs are severly limited in what they can do. It has been argued that there are hardly any useful strictly conforming C progams.

Non-strictly conforming programs are conforming with respect to a specific C implementation. It is then the job of the C implementation to define what programs it accepts beyond strictly conforming ones, and how it handles (or doesn't handle) undefined behavior. Everything is possible here, from the infamous DeathStation 9000 (with the most insidious UB behavior) to a fully deterministic C implementation that processes any program in the most unsurprising developer-friendly fashion.

There is arguabley a misconception that C is a single language. It is effectively rather a family of languages, for which the C standard only defines a common denominator.

For me, the canonical UB example is a buffer overflow. No matter how you define UB, in practice a buffer overflow can result in for example, a system crash or - given appropriate very specific input data - encrypting all the files on your hard drive for a ransom.

Requiring compilers to restrict UB to something similar to unspecified behavior (where the behavior is not specified by the standard, but a C implementation must pick a behavior and document its choice) would require C compilers to prevent arbitrary code execution given a buffer overflow in C code, i.e. ensure that a buffer overflow in C code does not ever result in actual buffer overflow on the physical machine. This seems implausible to achieve - given the tradeoffs chosen for C, once UB hits, it really is undefined as it can (not in all, but in some) scenarios result in executing absolutely arbitrary machine code depending on the data provided to the program.

On the other hand, if you'd just want to say that the compiler can't assume that UB won't happen for optimizations, then many optimizations are impossible because theoretical UB is literally everywhere where basic arithmetic happens because of the possibility of an integer overflow - if UB would be required to do "whatever the hardware does" then it means that you can't even reorder basic chains of arithmetic instructions and have to execute every operation in order as-written to have the appropriate behavior in case of overflow.

By the way, dereferencing NULL is a well defined behaviour on every computer architecture: you are basically reading at address 0 of memory. It just causes a crash if you have an operating system since it will cause a page fault, but in kernel mode or in devices without an OS is a legit thing to do (and even useful in some cases).

Why should C compilers make it undefined? The standard doesn't mandate that undefined behaviour should change the semantic of the program. Just define all the undefined behaviour that you could, to me keeping them undefined makes no sense (even from the standard point, everyone knows that if you overflow an int it wraps around, why should it be undefined??)

> The trouble was that in that context, a null pointer dereference didn't trap, (because it was kernel code? not sure.) so the bug was not detected.

Yes, because it was kernel code. Because that dereference is completely legal in kernel code. The C code was fine, assuming that it was compiled with appropriate kernel flags. This was not a bug in Linux, at least not on the level of the C code itself.

> The linux kernel now uses -fno-delete-null-pointer-checks to ensure that doesn't happen again.

I also seem to remember that it was already using other "please compile this as kernel code" flags that should have implied "no-delete-null-pointer-checks" behavior, and that the lack of this implication was considered a bug in GCC and fixed.

Yeah, I was about to post about this. Note that implementation-defined behavior and unspecified behavior are not _only_ used in places where a choice is given. Quite a few uses of implementation-defined behavior do not actually select between options. It just says the behavior is implementation-defined. So I suppose it selects between infinite options.

For this kind of behavior there is actually no limit on what the implementation is allowed to do, including assuming it never happens and optimizing accordingly. Implementations just need to document what they do.

I don't think these attempts to change the definition of UB are useful. As a compiler vendor it doesn't help to just say "it has a behavior", because that doesn't stop me from doing exactly what I do today. If people want some specific behavior, or to limit behaviors, then they need to actually say that in the spec.

To take left shift for example. Instead of saying it's implementation-defined behavior, they should say that it produces a implementation-defined non-trap value in the range of the resulting type that is consistent for the same inputs.

This would be pretty short to write in standardees, doesn't allow UB based optimizations, and allows the required implementation divergence.

> meticulously written, previously fine programs

With relatively few exceptions, if your program hits undefined behavior, then your program was already doing something pretty wrong to begin with. Signed overflow is a poignant example: in how many contexts is INT_MAX + 1 overflowing to INT_MIN actually sane semantics? Unless you're immediately attempting to check the result to see if it overflowed (which is extremely rare in the code I see), this overflow is almost certain to be unexpected, and a program which would have permitted this was not "previously fine" nor "meticulously written."

I feel compelled right now to point out that software development is a field where it is routine to tell users that it's their fault for expecting our products to work (that's what the big all-caps block of every software license and EULA says, translated into simple English).

If the program was previously "fine" on version x.y.z of some compiler, then it is most likely still fine on it. That's the target that the program was written for.

There's some disagreement on whether you can call a program "fine" that breaks after switching to a newer version, or a different compiler.

I see a lot of programmers out there that unfortunately use the behavior of their code on whatever compiler they're using at the moment as a proxy for what the language actually guarantees.

> the same place as suspicion of the "be lenient in what you accept" IETF principle

No. It's fine (arguably desireable, but reasonable people might disagree) for implementations that encounter undefined behaviour to terminate execution immediately, especially if it's with a error message. The problem is when implementations silently willfully misinterpret what they accept, particularly in ways that cause (not "expose"; reading from a null pointer and discarding the value (for example) isn't) security vulnerablities.

> optimizations that look at UB adversarially

The whole point is that there isn't such adversarial thing like "we're going to find the UB right there, won't even print a warning about it, and mess up your crappy code, haha!"

Optimizers aren't reasoning about code like people do (start to finish with high-level understanding of the whole function), but rather as series of mostly dumb, mostly isolated small passes, each pass changing one little thing about the code.

It just happens that one pass marks certain instructions as "can't happen" (like the spec says), then another pass simplifies expressions, and then another pass that removes code that doesn't do anything, usually left over from the previous steps. They sometimes combine in an "adversarial" way, but individually each pass is justified and necessary.

Compilers already have lots of different passes. Splitting optimizations into passes is a way to keep complexity closer to O(n) rather than O(n^2), but this architecture makes interactions between passes very delicate and difficult to coordinate, so it's difficult to instrument the data to avoid only cases of annoying UB without pessimizing cases that users want optimized.

Name one such optimization. We'll be happy to refute your points for that one.

Compiler dev here. Do you really think we just come up with code transformations and add them to the compiler just because?

All code transformations have a compile time cost and runtime perf impact. We don't add transformations unless the runtime perf impact greatly outweighs the compile time cost.

These optimizations are added because they measurably improve the performance of real code. This comes up in every review for new or updated optimization passes. This claim that they aren't justified is actually rather insulting to the effort put in to improve perf without taking days to compile.

What do you mean by "these UB optimizations"? C is a low-level language; it's basically impossible for a compiler to reason about the code unless it makes certain assumptions. It needs to assume the code is not self-modifying to do pretty much any code-generation more intelligent than a macro assembler. It needs to assume the code isn't messing with the stack frames/return addresses in order to inline functions. It needs to assume the code isn't using an out-of-bounds pointer to access a neighboring local variable, so that it can move local variables into registers. "gcc -O0" is a good approximation for the performance you get if the compiler isn't allowed to optimize based on UB.

Yes, that means C without optimizing without UB is slower than Java. Optimizations need some form of reasoning about what's happening. For Java it's optimizing based on guarantees provided by the language (there's no raw pointers that could mess with the things listed above). But C doesn't provide any hard guarantees, so instead it needs to blindly assume that the code will behave sanely.

Also note that for many of the more manageable sources of UB, most compilers provide a choice (-fwrapv, -fno-strict-aliasing, ...). Yet few projects use these options, even when they use other gcc/clang-specific features. Doesn't that indicate that C programmers indeed want to sacrifice safety for optimizations?

To do UB "optimizations", the compiler first needs to figure out that there is an UB it can "optimize" anyway. At this point instead of "optimizing" it could, and in my humble opinion absolutely should, blow up the compilation by generating an UB error, so people can fix their stuff.

What about backwards compatibility in regards to a new compiler version deciding to issue errors on UB now? You don't have any guarantees about what happens with UB right now, so if you upgrade to a new version compiler that generates errors instead of "optimizations" everything would be still as before: no guarantees. And it's frankly a lot better to blow up the compilation with errors than to have the compiler accept the UB code and roll a dice on how the final binary will behave later. You can either fix the code to make it compile again, or use an older "known good" version of the compiler that you previously used as a stopgap measure.

I fail to see any reason whatsoever why compilers are still doing all kinds of stupid stuff with UB instead of doing the right thing and issuing errors when they encounter UB.

I also fail to see why the C language designers still insist on keeping so much of the legacy shit around.

> But nobody has ever shown … that these UB optimizations actually improve performance of anything.

That’s just not true. Already examples in this thread: https://news.ycombinator.com/item?id=27223870

> I suspect so - makes it harder to reason about loop counts because the compiler can't necessarily guarantee that an incremented loop counter won't become negative and thus the loop needs to iterate more.

This is a favourite example that gets thrown around, but for all practical loops GCC and clang seem to have no problem even when you compile with -fwrapv

for (int i=param; i < param + 16; i++)

does not have a guaranteed loop count with the current rules. The loop body will execute 16 times if param <= INT_MAX-16, but if the expression "param + 16" can overflow, the behavior is undefined. (I'm assuming param is of type int.)

Assuming you defined signed integer overflow to follow two’s complement rules (the only reasonable interpretation other than UB), it would still be a guaranteed loop count of 16. (EDIT: i’m a dumbass, this is obvs not true. disregard this paragraph)

There’s an interesting thing to note with that example though: even if you did make signed integer overflow defined, that code is still obviously incorrect if param + 16 overflows. Like, the fact that signed integer overflow is UB is totally fine in this example: making it defined behavior doesn’t fix the code, and if making it UB allows the compiler to optimize, then why not?

Arguably, this is the case with the vast majority of signed integer overflow examples: the UB isn’t really the issue, the issue is that the programmer didn’t consider overflow, and if overflow happens the code is incorrect regardless. Why cripple the compilers ability to optimize to protect cases which are almost certainly incorrect anyway?

I don't know if there exists a C compiler that leverages this feature but there are ISAs (for instance MIPS) that can trap on signed overflow.

The fact that it's UB in C means that you can tell the compiler to generate these exception-generating instructions, which could make some overflow bugs easier to track down without any performance implications. And your compiler would still be 100% compliant with the standard.

That being said I just tried and at least by default GCC emits the non-trapping "ADDU" even for signed adds, so maybe nobody actually uses that feature in practice.

That's the exact reason why this rule was introduced into the standard: it was so C compilers could compete with Fortran compilers (Fortran has similar rules and at the time they were beating C compilers on equivalent scientific codes by 2-3x).

Fortran has even more restrictive aliasing rules than C: a function is allowed to assume that any two array arguments passed as arguments do not overlap. If they do, the behavior is undefined.

It's not so much about cleverness, but knowledge and vigilance. You first have to be aware of all the footguns, and then be careful not to let any of them slip through...

> UB means the compiler will trust me and concentrate on generate the fastest code ever.

In reality, UB means the compiler will assume it doesn't happen and work from there.

Of course a more expressive language could just make it so the compiler doesn't have to assume this e.g. a C compiler will consider a dereference as meaning the pointer is non-null, both backwards and forwards.

But if the language had non-null pointers, it would not need to bother with that, it would have a non-null pointer in the first place. It could still optimise nullable pointers (aka lower nullable pointers to non-nullable if they're provably non-nullable, usually after a few rounds of inlining), but that would be a much lower priority.

Expecting programmers to evaluate their own cleverness does not work. Every nontrivial C program has undefined behaviour, making it a security flaw waiting to happen - I've been in these kind of debates where a C advocate will claim that program X is correct, and literally every time it turns out that program X has undefined behaviour somewhere.

> e.g. removing a check for for overflow is definitely NOT ignoring the behavior. Deleting write because it would be undefined behavior for a pointer to point at some location is also NOT ignoring the behavior.

Depending on how you look at it, this is ignoring the behavior.

For example, say you have this:

    int f(int a) {
        if (a + 1 < a) {
            // Handle error
        }
        // Do work
    }

You have 2 situations:

    1. a + 1 overflows
    2. a + 1 does not overflow

Situation 1 contains undefined behavior. If the compiler decides to "ignor[e] the situation completely", then Situation 1 can be dropped from consideration, leaving Situation 2. Since this is the only situation left, the compiler can then deduce that the condition is always false, and a later dead code elimination pass would result in the removal of the error handling code.

So the compiler is ignoring the behavior, but makes the decision to do so by not ignoring the behavior. It's slightly convoluted, but not unreasonable.

How so? The implementation can, and perhaps should, define that it errors. Whatever behaviour you're worried about a compiler doing for implementation-defined behaviour, it could do exactly the same thing if the behaviour was undefined.

> In the C89 standard, undefined behavior becomes undefined UPON USE OF the thing that is undefined.

Is that still the case for current C standards, or did something change in C99/C11?

This is actually desired though, at least by some programs. For example, say you have a function with a very expensive loop that repeatedly performs a null check and then executes some extra code if it's null, but never sets the value. This is called from another function which uses the checked value without a null check (proving it's not null) before and after the loop ends. The first function is inlined. You want to tell the compiler not to optimize out the null check and extra code in the loop? Or that it can't optimize stuff out to reuse the value from the first use of the value? If so, what is the compiler allowed to optimize out or reorder?

Now, to see why this might actually produce a bug in working code--say some other thread has access to the not-null value and sets it racily (non-atomically) to null. Or (since most compilers are super conservative about checks of values that escape a function because they can't do proper alias analysis), some code accidentally buffer overflows and updates the pointer to null while intending to do something else. Suddenly, this obvious optimization becomes invalid!

Arguments to the effect of "the compiler shouldn't optimize out that loop due to assuming absence of undefined behavior" are basically arguments for compilers to leave tons of performance on the table, due to the fact that sometimes C programs don't follow the standard (e.g. forgetting to use atomics, or indexing out of bounds). While it's a legitimate argument, I don't think people would be too happy to find their C programs losing to Java in benchmarks on -O3, either.

> So, for example, undefined behavior that can be encountered within a loop makes it allowable to simply remove the loop. Even if the undefined behavior is inside of an if that does not happen to evaluate to true with your inputs.

The last sentence is not true. If there is UB inside the if, the compiler may assume that the if condition never evaluates to true (and hence delete that branch of the if), but it may certainly not remove the surrounding loop (unless it can also prove that the condition must be true).

> In current compilers, the existence of undefined behavior anywhere in your program is an excuse to do anything that the compiler wants to with all of the rest of your program. Even if the undefined behavior is never executed.

This is…complicated. Let's say you have an array of ten numbers, and then you take user input and use that to index into the array. This program is well-formed…as long as the user never inputs a number beyond ten. If they do, then the program is invalid. In general, the presence of undefined behavior is an attribute of the running program, not the source code itself. If there exists any execution where only defined behavior, the compiler may not deviate from the standard. However, what you probably meant is behavior in the face of the existence of runtime undefined behavior, in which case you are correct that a compiler could write clairvoyant code that refuses to execute the first instruction if it knows that UB will happen at some point in the program.

You could imagine bounds being baked into the types and checked at compile time.

    int{0..10}  foo  = 4
    int{0..10}  bar  = 5
    int{0..10}  buzz = foo+bar                // ERR: 10+10 potentially > 10
    int{0..10}  boom = wrapping_add(foo, bar) // OK
    int{0..100} sum  = foo+bar                // OK

There are tools which can do this, for example Code Contracts in C#. It becomes rather tedious and verbose so is something usually only left for very safety critical code.

Something that Ada, Modula and Pascal compilers are able to optimize away in most cases.

C keeps targeting an hardware model that even most Arduinos are super computers by comparisasion.

Same way it does in Javascript?

48-bit numbers. "The internal format of a single-precision floating-point number consisted of one unused bit, followed by the sign of the number, then the sign of the exponent, then a six-bit exponent, then 39-bit mantissa. The bias of the exponent was such that it could be considered to be in excess-32 notation as long as the mantissa was considered to be a binary integer instead of a binary fraction. This allowed integers to also be interpreted as unnormalized floating-point numbers."[1]

So there was only one numeric format at the hardware level. Convenient.

[1] http://employees.oneonta.edu/zhangs/csci201/general%20Floati...

> 1. write your own compiler that does it right (ETOOHARD now that clang didn't)

The problem here is that "right" is not quite that black-and-white. To a lot of users, "right" is "my code appears to work," optionally with "at a given performance level." A new compiler that's slower, or changes semantics compared to some baseline, won't necessarily be seen as "better" unless there's some clear benefit, and "doesn't perform optimizations that may result in security holes" wasn't necessarily a clear benefit at the time clang was trying to gain traction. "Better error messages" and "faster compiles", on the other hand, were, and those were some of the reasons Clang gained adoption despite (sometimes?) producing slower programs.

It's somewhat analogous to Microsoft's dedication to backwards compatibility. If a program did something technically illegal and a later version of Windows broke the program, users tend to blame Microsoft and/or Windows, not the program. Same thing here - if Clang didn't perform such aggressive optimization and was slower than GCC because of it, users will tend to think "Clang is slow", not "GCC is making nonsensical optimizations" or "My program invokes UB".

> The original intent was for signed overflow to be architecture-specific (not everything was 2s complement back then).

The term for that is implementation-defined, not undefined. If it were to be architecture-specific, back in C89, they would have used the term implementation-defined, as they do for things like the size of pointers.

There's "implementation defined" for the concept you describe.

That behaviour is no less undefined in the process of overflow.

> The idea that UB is carte blanche for implementations to do whatever is an unintended consequence of the vague language of the standard.

Whether or not this was originally intended, it's certainly become the way the standard is written and used today, so that's kind of beside the point.

Further, this is not some new idea that arose from the C standard. It's a basic, core idea in both software engineering and computer science! You define some meaning for your input, which may or may not cover all possible inputs, so that you can go on to process it without considering inputs that don't make sense.

Now, to be fair, the "guardrail-free" approach where UB is silent is a bit out of the ordinary. A lot of software that makes assumptions about its input will at least try to validate them first, and a lot of programming language research will avoid UB by construction. But C is in a unique place where neither of those approaches fully work.

> The C standard allows for both kinds of environments by stating that these behaviors are undefined, allowing the implementation to error out or do something sensible, depending on the environment.

This is true, but it doesn't mean that "something sensible" is actually something the programmer should rely on! That's just asking too much of UB- programmers need to work with the semantics implemented by their toolchain, not make up an intuitive/"sensible" meaning for their undefined program and then get mad when it doesn't work.

For example, if you want to scan through a bunch of memory, tell the language that's what you're doing. Is that memory at a fixed address? Tell the linker about it so it can show up as a normal global object in the program. Is it dynamic? Memory allocators fabricate new objects in the abstract machine all the time, perhaps your compiler supports an attribute that means "this function returns a pointer to a new object."

The solution is not just to shrug and say "do something sensible but potentially dangerous." It's to precisely define the operations available to the programmer, and then provide tools to help them avoid misuse. If an operation isn't in the language, we can add it! If it's too easy to mess up, we can implement sanitizers and static analyzers, or provide alternatives! Yelling about a supposed misreading of "undefined behavior" is never going to be anywhere near as effective.

> The C standards have the perfectly fine name "implementation dependent" to describe those things.

That term is not used by the C standards. Do you mean "implementation-defined"? asm is not among the explicitly specified implementation-defined behaviors, it's listed under "Common extensions". I don't see any mention at all of syscalls in C99. (I'm working with http://www.dragonwins.com/courses/ECE1021/STATIC/REFERENCES/... here.)

No. "Implementation defined" says "the standard doesn't specify what happens here but the compiler must document what it does". That's a step removed from "the compiler may define what this does".

Oh, for instance, on some implementations there is a lot of interesting stuff just prior to the allocated block returned. Not exactly the pinnacle of elegance but it gets the job done.

The C standard says very little about how those types work. In particular, you can cast a pointer to one of them and then cast back to a pointer -- but only if you cast the exact same value back, and the intptr values are not guaranteed to be in any way meaningful.

In particular, casting a pointer to intptr_t, doing arithmetic on it, and casting back is not guaranteed to do anything useful. It almost certainly will, since most systems treat it as roughly the same as casting to char *, but the standard does not guarantee it.

> The difference is that, once decided and written, no party (both old and new) can’t chime in with a new interpretation.

That double negative ("no party … can't") was accidental, right?

> Feel free to advocate for changing the standard accordingly.

I am, in fact, describing the existing C standard, not advocating for changes. Please refer to Annex L "Analyzability", which defines the terms I used: "bounded undefined behavior" and "critical undefined behavior". Note that this section is conditional... it is not widely adopted. I am advocating for increased adoption of this part of the standard, or barring that, revisions that would make adoption more palatable.

And in general, "If you don't like it, advocate changing the standard" is not a useful or insightful response. It is completely reasonable and normal to complain about something without trying to fix it.

Different code depends on different optimizations. A loop on an int** might benefit a lot from aliasing optimizations, because the compiler will assume that a[i] will remain the same after writing to a[i][j]. Other code may not benefit at all.

Likewise that loop may not benefit from signed overflow; instead an initialization loop that, by way of multiple level of macros, ends up doing a[i]=b[i]*1000/100, might become twice as fast if signed overflow rules let the compiler rewrite the assignment as a[i]=b[i]*10.

Wang et al tried this an experiment and found no serious wins.

I am sympathetic with the view that compiler writers are trapped by the architectures they have to support.

However, I think there's just a philosophical difference here about what compilers are for: are they responsible for emitting correct programs; or are they for literal transpiling?

Regardless of the history which has led to the latter outcome for C, I have to agree with the author that I think K&R's original vision wasn't quite so accommodating.

Perhaps ignorantly, I just have to imagine there is something a little better than the present attitude.

Could there not be a --hault-on-all-undef with a serious attempt to realise that in as many respects as possible?

> Like starting to optimize away loop checks that can "never happen" because signed integer overflow is UB, suddenly changing the behavior of programs that were fine for years?

Yeah. Not doing that on modern processors is actually quite disruptive.

Here:

    for(i = offset; i < (offset + 16); i++) {
        arr[i] = i + 32;
    }

What C compilers currently do is, in line with the standard, ignore the case that offset + 16 might overflow. This makes this eligible for loop unrolling, and depending on the specifics of the math inside the loop, the compiler can do a lot to pre-calculate things because it knows this is happening 16 times exactly.

If, instead, we force compilers to think about the fact that offset + 16 could have some implementation-defined meaning like wrapping, then all bets are off & we have to throw a bunch of optimization opportunities out the window.

Lots and lots of hot & tight loops which are currently able to be compiled into something suitable for the preferences of modern CPUs instead has to be to be naively compiled because of the need to hold back due to the possibility of something that largely wasn’t happening, happening.

Most people write most loops this way, never expecting or intending to overflow anything. Most loops are benefitting from this optimization. A lot of code would get slower, and programmers would have to do a lot more fragile hand-unrolling of operations to get that performance back. And they’d need to update that more often, as whatever the optimal “stride” of unrolling changes with the evolution of CPU pipelines.

It’s slower code and more work more often for more people, to satisfy a minority use-case that should really just have its own separate “please wrap this” construct.

C was created during a time where instructions were executed linearly with no vectorization, memory was a flat space with no CPU caches, and there wasn’t a branch predictor that may or may not execute the correct program branch in advance. The list goes on but the rest is beyond my scope.

C was designed for a now obsolete computer architecture model and over the years this old model has essentially become an abstraction that sits between C and the CPU. As such, C programmers aren’t really programming in their CPU’s domain anymore and C, by default, lacks the commands necessary to effectively utilize these new developments. It is left up to the compiler to translate C code from the old architecture abstraction into efficient machine code for our new machines.

For a more in depth look into this topic I recommend you check out (0).

(0) - https://queue.acm.org/detail.cfm?id=3212479

It would be nice if that were true, but it's not. The ability to assume incrementing a counter won't overflow makes range analysis possible in many cases where it otherwise wouldn't be. Because of this you can perform vectorization because the compiler can assume it knows how many times the loop will run.

The performance differences are not small.

You can also tell some compilers to treat signed integer overflow as wrapping-- but people don't usually do this because the optimizations you lose are valuable.

There is no guarantee that an addition in C is even going to convert into an add instruction, though. Actually, on x86 it's more likely to turn into a lea because this optimizes port usage better.

Another thing is that the wording around some of the UB issues is just plain bad. The most extreme probably is the rules around strict aliasing. That there, for quite a while, was uncertainty whether the rules allow type punning by reading a union member when the last write was to another member of a good example of not taking reality into account. Yes memcpy exists - but it is even less type safe!

See e.g. https://github.com/postgres/postgres/blob/8bdd6f563aa2456de6...

You're right, it should be automatic storage duration instead of static storage duration.

The first code isn't necessarily buggy.

As a rule of thumb, yes, compilers should issue warnings where undefined behavior is obviously occurring. (And there is something to be said for compiling with -Werror). However, that's not going to always work, and there are two reasons for this.

The first reason is that undefined behavior is ultimately a statement about dynamic execution of the program. The set of expressions that could potentially cause undefined behavior is essentially all of them--every signed arithmetic expression, every pointer dereference, hell, almost all function calls in C++--and figuring out whether or not they actually do cause undefined behavior is effectively impossible at the compiler level. This is why sanitizers were developed, and also why sanitizers only work as a dynamic property.

For a concrete example, consider the following code:

  extern void do_something(int * restrict a, int * restrict b, size_t n);

  void my_function(int *y, int *z, size_t x, size_t n) {
    if (x > n)
      do_something(y, z, n);
  }

This code could produce undefined behavior. Or it could not. It depends on whether or not y and z overlaps. Maybe the check of the if statement is sufficient to guarantee it. Maybe it's not. It's hard to advocate that the compiler should warn, let alone error, about this kind of code.

The second issue to be aware of is that there is a general separation of concerns between the part of the compiler that gives warnings and errors (the frontend), and the part that is actually optimizing the code. It is difficult, if not impossible, to give any kind of useful warning or error message in the guts of the optimizer; by the time code reaches that stage, it is often incredibly transformed from the original source code, to the point that its correlation with the original can be difficult to divine.

So I once came across some really weird code that broke an optimization pass I was working on. It looked roughly like this (approximate C translation of the actual IR):

  if (nullptr != nullptr) {
    int *x = nullptr;
    do {
      /* do some stuff with x */
      x++;
    } while (x != nullptr);
  }

What hideous code creates a loop that iterates a pointer through all of memory? Why, this (after reducing the test case):

  void foo() {
    std::vector<std::set<int>> x;
    x.emplace_back();
  }

So the crazy code was generated from the compiler very heavily inlining the entire details of the STL, and the original code was a more natural iteration from a start to an end value. The compiler figured out enough to realize that the start and end values were both null pointers, but didn't quite manage to actually fully elide the original loop in that case. Warning the user about the resulting undefined behavior in this case is completely counterproductive; it's not arising from anything they did, and there isn't much they can to do to silence that warning.

> This is permissible (on typical hardware)

Hardware doesn't define C semantics. C defines C semantics. I don't understand what your conditional example is trying to say, but you seem to assume that an uninitialized y is an actual uninitialized register or memory location. It's not.

>> A function marked _Noreturn returns.

> If written in C, the body of the function must include a return statement, and is thus a compile time error.

It might contain a return statement that is never reached because the function goes into an infinite loop or exits/aborts the program or does a longjmp.

I believe he’s arguing that they were more sane/pragmatic compilers — they would be inherently less comfortable exploiting UB to do anything other than what people expected or were used to, because there is more real possibility of retribution (GCC could get away with making demons fly out your nose and just lose marketshare [that it doesn’t directly depend on anyways] where a commercial compiler would go out of business)

I see this argument a lot, but it's silly. I don't have to care whether gcc changed the settled semantics without notice or documentation out of sincere belief that they were helping or out of a desire to beat a benchmark that doesn't matter to me, or out of habit. It's not the intent of the compiler authors that matters, but their disregard of the needs of application programmers.

> Note that if the standard had not declared "n << 32" to be all-bets-are-off UB, but instead had said something like, "it results in some implementation-specific value, or maybe traps" -- as a rational person would presume -- then this would not turn into a security problem.

But also note that a lot of existing code doing bitshift-and-index inside a hot loop that never went out of bounds would now get slower if it started having to run bounds checks it had previously elided in an optimization pass.

Let's not pretend that "it results in some implementation-specific value, or maybe traps" is a clear win with no downsides that Standards Authors and Compiler Engineers are ignoring out of some kind of malice – there are very real performance tradeoffs here, and a new version of the standard that makes a lot of existing real-world code slower isn't going to be a popular one with many people.

I think the compiler that you were using is broken. One can't infer "index < 32" unless on a code path to a use of "flag", and that inference can't override the predicates that dominate that use.

That's such a good example for teaching purposes, because it manages to combine the lack of understanding surrounding the remainder operator (i.e. unlike python % in c is not modulus) with the lack of understanding surrounding 0x80000000 (two's complement bane) into a single example. However it still makes my point that in this circumstance, the compiler's strategy still is to do nothing, because it can prove that the check could only be true under undefined behavior circumstances, so doing nothing means not compiling the check. I'm fine with that. Would anyone prefer that the compiler's internal definition of logic assume that absolute values can be negative? Must we throw out centuries of math because we've optimized integers to be 32-bits?

The only thing that's problematic is we need better tools to bring logic assumptions to our attention. Currently, UBSAN can only warn us about that when two's bane actually gets passed to the function at runtime. So the only way to spot faulty logic we failed to consider is to both enable UBSAN and be super methodical about unit testing.

Well, another thing I like to do is just read the assembly output. Constantly. Whenever I write something like a parser I've got a keyboard mapping that shows me the assembly output in Emacs with UBSAN enabled. If I turn off the noisy ones like pointer overflow then I can avoid these issues altogether by writing my code so that no UBSAN assembly gets generated. Since the compiler won't show that as a warning. You literally have to read the -S assembly output to get the compiler warnings that are actually meaningful.

I wish there was an abs() function that returned the unsigned version of the signed input type and wasn't UB if you passed in the most negative value of the signed type. Sadly neither C or C++ nor Rust's standard libraries have a built-in implementation of this, and sending the most negative integer into Rust's .abs() will panic in debug mode (is not intended to be used in properly implemented code).

What you've written doesn't demonstrate the issue outlined in the big report. The issue is that real-world code defends against certain hostile inputs like so, and that the optimization breaks those defenses:

    int a,b,c;
    a=INT_MAX;     /\* statement ONE */
    b=a+2;         /* statement TWO */
    c=(b>a);       /* statement THREE \*/

Whether or not you think this is technically permissible by a sufficiently self-serving (from the perspective of a person whose ONLY goal is speed optimization) reading of spec is irrelevant. Any reading of the spec should err on the side of protection against real-world consequences.

I don't want lawyering and technically correct-ing, I want pragmatism.

Just like this one, when I posted mine here

Wow! Interesting to see hints that meltdown exists years before it was officially published.

> This exists, but the effect of undefined behavior in CPU architectures is a little bit more forgiving than the interpretation of UB in C to mean "literally the entire program has no meaning".

That is not quite what the interpretation of UB in C is, AFAIK. UB in C is generally interpreted as meaning that any path which would trigger an UB is invalid, because if it will be invalid once the UB is reached, well, if we know for sure we're going to UB for the instruction before it, we can say that we're already in UB.

Whole-program invalidity can occur when the compiler manages to infer that no execution path is UB-free, in which case yes the program is meaningless. More generally, programs will go off the rail as far ahead of the UB as the compiler managed to determine that the UB would unfailingly be reached.

And it's usually because the compiler works backwards: if a path would trigger an UB, that path can not be legally taken, therefore it can be deleted. That's why e.g. `if (a > a + 1)` gets deleted, that expression makes sense if you assume signed integers can overflow, but the compiler assumes signed integers can't overflow, therefore this expression can never be true, therefore it's dead code.

This is important, because many such UBs get generated from macro expansion and optimisations (mainly inlining), so the assumption of UB-impossibility (and thus dead code) enables not just specific optimisations but a fair amount of DCE, which reduces function size, which triggers further inlining, and thus the optimisations build upon one another.

The situation is different, because a CPU is by definition an interpreter. It does't perform code transformation, at least not at a higher level as a compiler. The CPU only looks at the next few instructions and perform them. A compiler, however, is responsible for taking a large coding unit and produce a transformation that is efficient. That process requires thinking about what is invalid code and operate on that.

There actually is a fair amount of truly undefined behavior for CPUs, but it's always at system/kernel mode rather than userspace for security reasons. You can search an ARM ISA for "UNPREDICTABLE" to see examples.

I’m not willing to accept a “say” for performance here.

I’d also want performance impact on a non-benchmark perf sensitive code base (browser, database, etc rather than spec).

The problem with the claim that the code is incorrect is that it is only* incorrect due to the imo bogus interpretation of what should be UB.

If I write

    If (a+b<a)

On any modern architecture that has a well defined and understood meaning. But compilers have repeatedly overridden I well understood behavior in the name of perf benefits that only turn up in Spec-style benchmarks

NPE isn't really a relevant comparison point - it raises an exception of sorts in both languages. Accessing an array past its end, accessing freed memory, reading uninitialized memory seem more apt.

"Undefined behavior" is a term of art relevant to C, meaning that the standard no longer has any comment about what happens. Thus the comments about launching nukes, or destroying your machine, etc., being standards complaint, even though obvious real compilers won't actually emit code that does that.

Dereferencing a nil pointer in Java is not undefined behavior. It is defined; it throws a NullPointerException, and "it throws a NullPointerException" is a very specific term of art in the Java community that is very precisely defined.

Many languages don't have an explicit concept of "undefined behavior" in their standard (though, of course, due to deficiencies in the standard they may have implicitly undefined behavior), and of those that do I doubt anyone matches C or C++ in the commonly-used category of languages.

Rust inheriting it is the equivalent of a codegen bug, and they get patched. The cause here isn’t LLVM, it’s language semantics.

Name a feature of C that makes it inherently "faster" than another compiles-to-machine-code language. I can name plenty that make it slower.

I don't really like Rust; I like C. But, it could be improved, making one with less undefined behaviour and better macros and less confusing syntax for types etc. Many of the new programming languages that try to avoid the bad thing from C I think are avoiding most of the good thing from C, too.