Hacker News Reader

Comment by AlotOfReading | Hacker News Reader

I wish UB were only as nasty as "nondeterministic behavior". In fact, if there's UB in anything the compiler sees, nothing at all can be assumed, including whether you even get an output. What you've given the compiler isn't C, so it doesn't have any obligations to do anything with it. The codepath with UB doesn't have to run for the nuclear rockets to launch and the nasal demons to appear.

Since approximately every nontrivial program ever written has UB, in actual practice we're only saved by the fact that compilers aren't entirely maliciously compliant.

View on HN

That's not true. If the program's execution path from start to finish avoids UB then you're safe. (Also the source code itself has to avoid UB, but that part isn't hard.)

It's true that code with UB does not have to be reached, per se, but it does have to be something your program will reach before it can hurt you.

You're correct in practical terms, but I'm making a very pedantic point about what the standard requires happen, mainly because this pedantry has important implications for e.g. safety critical C. Note 1 to the definition in 3.4.3 provides some clarification about the extent of UB and states that UB can manifest at translation time. It also gives says that the translator should behave in a documented manner when encountering UB, but does not require that it do so.

C has both translation-time UB and runtime UB. (C++ explicitly separates the two concepts into "ill-defined, no diagnostic required" and "undefined behavior".) You can tell them apart from the condition for UB to occur: if it's a translation-time condition, then it's translation-time UB, and if it's a runtime condition, then it's runtime UB. (Same with implicit UB: is it a translation-time or a runtime assumption being violated?)

Usually when we talk about UB, we're implicitly talking about runtime UB, since translation-time UB is generally far less subtle. If a program contains only conditional runtime UB, the compiler is not permitted to break the entire program from the very beginning, since all possible executions that do not trigger runtime UB must execute correctly as per 5.1.2.3.

5.1.2.3 only binds conforming programs. Programs containing UB are by definition non-conforming.

I hadn't considered the C++ standard here, but 1.9 is much more clear than corresponding C verbiage. 1.9.5 is exactly what's described upthread, where any "execution [that] contains an undefined operation" has no prescribed behavior. But the note to the requirement immediately before that (1.9.4) doesn't use that language and instead "imposes no requirements on programs that contain UB". If they had intended only to avoid specifying semantics for programs that hit UB during some possible execution, they would have used the same language as 1.9.5.

Your claim is actually false. C differentiates between a conforming program and a strictly conforming program. 5.1.2.3 binds to conforming programs which is permitted to produce output dependent on undefined behavior.

Only strictly conforming programs may not produce output dependent on undefined behavior.

No? Conformance allows unspecified and implementation defined. Strict conformance is the absence of that (i.e. same output in every conforming environment). Neither includes UB, as UB is "outside the standard" in some sense and doesn't have defined semantics.

It's a common misconception that a conforming program may not engender undefined behavior. In fact this very article touches on how realloc has introduced new (and backwards incompatible) undefined behavior precisely to accommodate the POSIX standard (so that POSIX compliant implementations of C can redefine the otherwise undefined behavior however they please).

Can you cite that? It runs against a plain reading of the standards (both C and C++) and would be insane for the standard to allow "correct" programs to include those with undefined behavior. There was even an unadopted proposal (n853 [1]) attempting to clarify this.

While I was making sure I wasn't missing something obvious, I took a look through the rest of the WG14 proposals to see if I was somehow off in my understanding regarding translators being allowed to barf over UB anywhere in the program. There was a proposal clarifying the situation to the possible-execution understanding from upthread submitted by Victor Yodaiken (n2278 [2]), but unfortunately it was also never adopted.

[1] https://www.open-std.org/jtc1/sc22/wg14/www/docs/n853.htm

[2] https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2278.pdf

You are now mixing correct and conforming programs. Neither the C or C++ Standard mention anything involving correct programs. C uses the term conforming program and C++ uses the term well-formed program.

In C a conforming program is any program that satisfies any single conforming implementation, even if said implementation includes extensions or non-portable constructs. A strictly conforming program is a program that satisfies every conforming implementation, which implies that said program does not produce an output that depends on undefined, unspecified, or implementation defined behavior.

In C++ a well-formed program is any program that satisfies the syntax rules, diagnosable semantic rules and the one-definition rule.

Going off of the C17 numbering,

4/7: "A conforming program is one that is acceptable to a conforming implementation."

This definition has no restrictions regarding runtime requirements, unlike for strictly conforming programs.

5/1: "An implementation translates C source files and executes C programs in two data-processing-system environments, which will be called the translation environment and the execution environment in this International Standard. Their characteristics define and constrain the results of executing conforming C programs constructed according to the syntactic and semantic rules for conforming implementations."

So clause 5 binds all "conforming C programs constructed according to the syntactic and semantic rules for conforming implementations", not just strictly conforming programs.

Now, 4/3: "A program that is correct in all other aspects, operating on correct data, containing unspecified behavior shall be a correct program and act in accordance with 5.1.2.3."

We can interpret this as saying that "a program that is correct in all other aspects... containing unspecified behavior" is "constructed according to the syntactic and semantic rules for conforming implementations" even if it only works when "operating on correct data".

From there, it does not seem very difficult to conclude that in general, a conforming program which contains fully specified behavior, assuming it operates on correct data, is also "constructed according to the syntactic and semantic rules for conforming implementations", and is therefore bound by clause 5. If we were to instead take the negation of this conclusion, that a program is not bound by clause 5 if any possible input data causes it to violate a runtime requirement, then the wording of 4/3 would not make any sense.

(In other words, every conforming program has a corresponding set of "correct input data", and it is correct and bound by clause 5 if it does not violate any runtime requirements when given any input data within that set. A program is only incorrect if that set is empty, i.e., the UB is unconditional.)

---

Meanwhile, I suppose you're looking at C++17. The note in [intro.execution]/4 is non-normative, and all of the normative language (e.g., on the very next paragraph) attaches runtime UB to the execution as a function of the input data, not the pure program.

[intro.compliance]/(2.1) and its (non-normative) footnote further clarify the distinction, stating, "If a program contains no violations of the rules in this International Standard, a conforming implementation shall, within its resource limits, accept and correctly execute that program.... 'Correct execution' can include undefined behavior, depending on the data being processed; see 1.3 and 1.9." This suggests that a program that executes undefined operations does not necessarily contain any rule violations.

I think your confusion here is coming from the fact that "unspecified behavior" is a specific thing in the standard's terminology (looking specifically at n3088 right now), distinct from the concept of "undefined behavior". So when it says "constructed according to the semantic rules", that inherently excludes UB, which definitionally has no semantics prescribed for it (unlike unspecified behavior). For brevity's sake, I'm ignoring the allowance that an implementation can give any particular UB defined semantics.

To reduce my point to a list of options:

* No UB in the program -> Specified by 5.*

* No UB for certain inputs -> Specified for those inputs, not specified otherwise

* UB present, but not on any possible execution path -> Not specified (this is the argument)

* UB present on every possible execution path -> Not specified (definitionally)

I linked this in a sibling comment, but there was a proposal to amend C2x's wording here to specifically exclude this type of insanity (n2278), but it wasn't adopted because it could potentially prohibit optimizations and the working group doesn't really want to address the issue of undefined behavior with more definitions.

My claim is, "If a program violates the semantic rules at runtime given input A, but does not violate the semantic rules at runtime given input B, then the execution of the program given input B will be defined by clause 5." This is because the wording of 4/3 implies that "being constructed according to the semantic rules" is a function of both the program and the data it is given, such that the behavior can be defined on some inputs by clause 5, but undefined on other inputs.

As I understand it, your claim is, "If a program violates the semantic rules at runtime given input A, then the behavior of the program is undefined given input B, even if the program would not have violated the semantic rules at runtime given input B." Am I misunderstanding your claim?

> I linked this in a sibling comment, but there was a proposal to amend C2x's wording here to specifically exclude this type of insanity (n2278), but it wasn't adopted because it could potentially prohibit optimizations and the working group doesn't really want to address the issue of undefined behavior with more definitions.

N2278 seems entirely irrelevant to this question, of whether potential UB given one input can cause unexpected behavior given another input. Instead, it seems to say, "If the program violates the semantic rules at runtime given input A, causing UB, then the implementation is forbidden from making that behavior identical to the program's hypothetical behavior if it had been given another input B." That looks pretty unworkable in the general case. (E.g., must compilers operate as if an out-of-bounds write on one object can modify the value of a totally different object?)

I realized on thinking a bit more that two of the cases I mentioned are actually identical, so you're correct. Good to know going forward!

Fine. HN is, after all, a place where you can be pedantic.

But those of us who are actually writing programs mostly care about "in practical terms", and in practical terms, this doesn't happen, so we don't care. We've got enough trouble worrying about what does happen; we don't have time and energy to worry about what doesn't and won't happen.

To provide some more context/motivation for why you might care, I write safety-critical code. I'm often advising people what they need to do for certification, etc. If all you need to do is ensure that you never execute undefined operations and knock out the list of specified UB, that's totally, 100% manageable. Throw some sanitizers on, provide realistic input, and test the hell out of it. Normal stuff.

If the reality is that any UB can invalidate the entire program (as is the interpretation taken by other standards re: C), then that's not remotely sufficient. You have to ensure the complete absence of UB.

That's like saying: "I don't care what the standard says!"

Sure, this is perfectly fine.

Only that you're not writing any C/C++ than, but something in the "gcc 12 language with some switches", or maybe the "LLVM 15 language with some switches", or something like that.

Well, if Visual Studio (or whatever Microsoft calls their compiler these days), and all known versions of gcc, and all known versions of LLVM all do something sane, then I'm not sure I care all that much about the theoretical possibility that some compiler someday might do something insane.

> approximately every nontrivial program ever written has UB

You can replace "UB" for "bugs" and the result is the same. UB is a bug on the part of the programmer, from the point of view of C, similar to dereferencing a null pointer. When the standard says that something is UB, it is just clarifying what these situations are.

That's exactly why a compiler shouldn't be able to 'optimize' in the face of UB, it should be an ERROR and the section of undefined behavior highlighted in the error message.

This would mean you’d have to insert a check every time you add two signed integers together, because signed overflow is UB. You’d also have to wrap every memory access with bounds checks, because OOB memory access is UB.

There are also tons and tons of loop optimizations compilers do for side-effect free loops which would have to be removed completely. This is because infinite loops without side effects are UB. So if you wanted these optimizations you’d have to prove to the compiler — at compile time — that your loop is guaranteed to terminate since it is not allowed to assume that it will. Without these loop optimizations, numerical C code (such as numpy) would be back in the stone ages of performance.

Edit: I just wanted to point out that one of the new features in C23 is a standard library header called <stdckdint.h> that includes functions for checked integer arithmetic. This allows you to safely write code for adding, subtracting, and multiplying two unknown signed integers and getting an error code which indicates success or failure. This will be the standard preferred way of doing overflow-safe math.

Another option would be to define behaviors for integer overflow and out of bounds memory access. Presumably they happen fairly often and it might be a good idea to nail down what should happen in those cases.

Those things aren’t up to the language, they’re up to hardware. C is a portable language that runs on many different platforms. Some platforms might have protected memory and trap on out of bounds memory access. Other platforms have a single, flat address space where out of bounds memory access is not an error, it just reads whatever is there since your program has full access to all memory.

The same goes for integer overflow. Some platforms use 1’s complement signed integers, some platforms use 2’s complement. Signed overflow would simply give different answers on these platforms. The standards committee long ago decided that there’s no sensible answer to give which covers all cases, so they declared it undefined behaviour which allows compilers to assume it’ll never happen in practice and make lots of optimizations.

Forcing signed overflow to have a defined behaviour means forcing every single signed arithmetic operation through this path, removing the ability for compilers to combine, reorder, or elide operations. This makes a lot of optimizations impossible.

The problem is that here is a vicious circle.

Most old computer architectures had a much more complete set of hardware exceptions, including cases like integer overflow or out-of-bounds access.

In modern superscalar pipelined CPUs, implementing all the desirable hardware exceptions without reducing the performance remains possible (through speculative execution), but it is more expensive than in simple CPUs.

Because of that, the hardware designers have taken advantage of the popularity gained by languages like C and C++ and almost all modern programming languages, which no longer specify the behavior for various errors, and they omit the required hardware means, to reduce the CPU cost, justifying their decision by the existing programming language standards.

The correct way to solve this would have been to include in all programming language standards well-defined and uniform behaviors for all erroneous conditions, which would have forced the CPU designers to provide efficient means to detect such conditions, like they are forced to implement the IEEE standard for floating-point arithmetic, despite their desire to provide unreliable arithmetic, which is cheaper and which could win benchmarks by cheating.

CPU designers don't like having their hand forced like that. If you create a new standard forcing them to add extra hardware to their designs, they'll skip your standard and target the older one (which has way more software marketshare anyway). They will absolutely bend over backwards to save a few cycles here and a few transistors there, just so they can cram in an extra feature or claim a better score on some microbenchmark. They absolutely do not care at all about making life easier for low-level programmers, hardware testers, or compiler writers.

I don't believe adding simple checks against data already present in L1 caches and marked as "unlikely to fail" should be so onerous.

> In modern superscalar pipelined CPUs, implementing all the desirable hardware exceptions without reducing the performance remains possible (through speculative execution), but it is more expensive than in simple CPUs.

Yeah, and that's how you get security vulnerabilities!

doesn't C force 2s complement now? If so, one less thing to worry about.

Good luck defining the behaviour of use after free of accessing out of bound stack memory without bound checking and GC.

UB is a better option though. When your signed integer overflows it's a bug nevertheless. Why force the compiler to generate code for a pointless case instead of letting it optimize the intended one?

If you value never having bugs over performance then just insert a check or run your program with a sanitizer that does that for you. It's a solved problem for a case where performance doesn't matter. The thing is that it does.

That would be great if it was possible, but how do you specify & implement sensible behavior for this:

    void foo(int *a, int b) { a[b] = 1}

At runtime there is no information about whether that write is in bounds and no way to prevent this from corrupting arbitrary data unless you compile for something like CHERI.

In checked languages this would probably be an 'unsafe' function, since it lacks those features.

If this were accessible at build time it could be checked for anything that references the function and bounds checked accordingly.

The promotion of a pointer to an array is really the source of the logical error. A language could place range checks on created arrays, and pointers / references to allocated arrays could be handled differently than anonymous slabs of memory. However an array without bounds (even stored elsewhere from just before the array's starting address) is as unsafe as 'null terminated strings' for length bounds. That's an idea that made much more sense when systems were much smaller and slower and the exposure to untrusted code and data were also far lower.

void foo(void *a, int b) { (int[])(a) = 1 } // Not quite C pseudocode, also see poke()

They don't happen that often. That's why they're bugs!

> because signed overflow is UB

no longer

> you’d have to insert a check every time you add two signed integers together,

This is exactly what is done in serious code. It is typically combined with contracts and static analysis (often human), e.g. "it is guaranteed that this input is in range 10-20, so adding it with this other 16 bit int can be assumed to be below sint32_max".

Great, those checks can stay in "serious" code, and those of us who don't want them can take the UB. C++ 20 actually ended up specifying that all ints are twos complement, removing this from the category of "UB," but a lot more weird stuff is programmed in C.

Note that signed overflow is still UB in c++ even with 2-complement being guaranteed for signed types.

We rehash this argument every few weeks. Please search the comment history why it is nonsensical.

Doing that at compile time would require being able to perfectly predict everything the program can do, which is equivalent to solving the halting problem (make the program do something undefined after it finishes, then if you get an error at compile time then it halts) and is mathematically impossible. Doing it at runtime would have a massive performance impact

What the standard explicitly calls out as UB is only a small subset of actual UB.

While you can certainly classify all UB as "bugs", doing so misses the critical differences between UB and other categories of bugs. If you have a logic bug for example, your program will correctly and consistently do the wrong thing. It will continue doing that wrong thing with a different compiler, on a different platform today and 10 years from now. Implementation defined behavior is a bit looser, but will still be consistent with any particular implementation (which will document the behavior) and will only manifest in the code that depends on it. A PR inserting one of these "normal" bugs doesn't invalidate the entire rest of the program.

UB is different. You can't make assumptions about UB because from the point of view of the standard, UB is "not C". There are no assumptions to be made, it's just all the stuff that doesn't have assigned semantics. And since the input is meaningless, so is the entirety of whatever the compiler gives you back.

> If you have a logic bug for example, your program will correctly and consistently do the wrong thing.

Not correct. Bugs can occur differently in different architectures, even in high level languages. UB is just a kind of bug whose effect depends on how the compiler behaves, so you have to be careful to test your code on different compiler settings. This is nothing new on programming languages, it is only made explicit in the C standard. Suddenly people started to believe that pointing out the obvious source of bugs (UB) in the standard is equivalent to let programs misbehave.

> you have to be careful to test your code on different compiler settings.

The problem is you have to test your code on compilers that don't exist yet with compiler settings that do different things from any compiler that ever might exist.

This has always been the case. If you write code that has UB, new compilers can do something yet undefined, by definition.

I'm not sure if you're making a point about "unspecified behavior" (where the compiler can choose between multiple valid behaviors), but no, a strictly conforming program will have the same semantics on different architectures. Strictly conforming programs can still have bugs, but their nature is completely different than UB because that's the point of the standard.

If they are bugs they should be reported to the user and end the compilation with an error.

Compilers actually have some options to enable that.

The problem is, it only works well in the simplest cases when the code will 100% exhibit UB within a single function.

In most cases, the UB would only manifest on particular input values - if you want your compiler to warn about that then it will report one "potential UB" for every 10 lines of C code, and nobody wants to use such a compiler.

Bugs are UB-like in a sense (what's the code going to do? well, you'll have to think about it, or try it and see), but UB is strictly worse than bugs (different compilers, even different versions of the same compiler, can do radically different things way beyond the scope of the bug).

Reply on news.ycombinator.com