How GCC and Clang handle statically known undefined behaviour
diekmann.uk
diekmann.uk
However, in regular code UB may only become known after multiple optimization passes (e.g. constant propagation, inlining).
It's difficult for the optimizer to meaningfully err at that time. The code may have been heavily modified by other optimization passes, which can add UB to the code (e.g. UB may have been data-dependent (1/x), but optimizer added a specialized copy of a function for x=0 where it became unconditional). The IR may have been generated by a non-C language that used UB/poison intentionally to guide optimizations.
I don't need static warnings that the compiler refused to do what I asked it to.
I need the 'UB means arbitrary transforms of your code with a bias towards deletion' invention to go away. It's a dubious premise in C++ where it supports people writing clumsy application code. It's totally inappropriate for things like implementing a JIT.
The reason we need low-level systems stuff is to bit-bang on hardware; put data in hardware formats; look at the low-level representation of things that we "shouldn't"--like integers, floats, objects, structs, stack frames; and implement the language runtime itself. The intent is usually pretty clear from what's written, and the mental model of the systems programmer is "I know I need to make the hardware do this thing". Compilers optimizing away UB and otherwise treating it as never going to happen actively get in the way. UB makes a systems language not a systems language anymore.
See for example: https://github.com/titzer/virgil/blob/master/doc/tutorial/Ra...
If you "just" want introspection, that's a little bit more reasonable; in principle something like dwarf unwind info could be extended for that. But, a) that a lot of work for an extremely niche problem and b) there is no guarantee that the in-memory representation of objects is stable at instruction boundaries, I think you would need something like GC write barriers.
So the ABI allows you to do this sort of stuff reliably as long as the compiler isn't doing inane things with UB.
If you simply meant that you want rely on an ABI, then that's fine, although relying on those details might be UB for standard conforming code, it is obviously well defined for compilers that conform to the ABI. Just because it is undefined from a standard point of view, doesn't mean is still undefined for a specific compiler + platform.
You will still need to use compilation firewalls and barriers to avoid aliasing or pointer provenance issues of course.
GCC leaves the print there because it must. While undefined behaviour famously can time travel, that’s only if it would actually have occurred in the first place. If the print blocks indefinitely then that division will never execute, and GCC must compile a binary that behaves correctly in that case.
I've always been told that the presence of UB in any execution path renders invalid all possible execution paths. That is, your entire program is invalid once UB exists, even if the UB is not executed at runtime.
Are you saying this isn't quite true?
If you do `5 / argc`, that's only undefined behavior if your program is called without any arguments; if there are arguments then the behavior is well defined.
Instead, the presence of UB in the execution path that is actually taken, renders invalid the whole execution path (including whatever happens "before" the UB). That is, an execution path has either defined or undefined behavior, it cannot be "defined up to point-in-time T". But other execution paths are independent.
Thus, UB can "time-travel", but only if it would also have occurred without time travel. It must be caused by something happening at runtime in the program on the time-travel-free theoretical abstract machine; it cannot be its own cause (no time travel paradoxes).
So the "time-travel" explanation sounds a lot more scary than it actually is.
Edit: argv is the name, argc will be 1
void f()
{
int d;
bool divide;
std::cin >> d >> divide;
std::cout << (divide ? 1/d : d);
}It is not. The presence of UB in an execution path renders that execution path invalid. UBs are behaviours, essentially partial functions which are allowed to arbitrarily corrupt program state rather than error.
However "that execution path" can be extensive in the face of aggressive advanced optimisations.
The "time travel" issue is generally that the compiler can prove some paths can't be valid (they always lead to UB), so trims out those paths entirely, possibly leaving just a poison (a crash).
Thus although the undefined behaviour which causes the crash "should" occur after an observable side-effect, because the program is considered corrupt from the point where it will inevitably encounter an UB the side-effect gets suppressed, and it looks like the program executes non linearly (because the error condition which follows the side effect triggers before the side effect executes).
The UB doesn't actually need to be encountered, just guaranteed to be encountered eventually (in a non-statistical meaning), that is where the time travel comes from e.g. if you have
if (condition) {
printf("thing\n");
1/0;
} else {
// other thing
}
if (condition) {
// crash
} else {
// other thing
}
// other thing
That's what I meant. Anything that leads to UB itself has UB.
In theory, the compiler could know that since `printf` is a well-known standard function. In practice, `printf` might even exit the program via SIGPIPE, so I don't think any compiler will assume that it definitely will return.
Not just sigpipe. Getting sigttou is part and parcel of how command line apps share control of the tty.
Division by zero is undefined behavior. The compiler can assume that it will not happen.
If the divisor is not zero, then the calculation has no side effects. The compiler may reorder the division above the print, because it would have no observable difference in behavior. This could be useful because division has a high latency, so it pays to start the operation as soon as the operand values are known.
If the divisor is zero, the UB says that there is no requirement on how it's compiled, so reordering the division above the print is legal.
Division has no side effects, and division by 0 is UB. UBs only occur in invalid programs, so behaviour in case of UB is not relevant to a discussion of side effects or their lack thereof, in language terms these are not programs at all.
const int div = 0;
if(div) {
return 1/div;
}
return 0;
If you replace 'if(div)' with an opaque function call, that doesn't change anything as the function might never exit the program, never return, long jump or return via an exception.
Is `printf` allowed to loop infinitely? Its behaviour is defined in the language standard and GCC does recognize it as not being a user-defined function.
"undefined behavior - behavior, upon use of a nonportable or erroneous program construct or of erroneous data, for which this document imposes no requirements"
It says "for which" and not "for the whole program". So the "interpretation" that a program becomes invalid or other requirements are affected was never supported by the text in the C standard. This was justified by some with arguments such as: "no requirements" includes behavior that can go back in time or otherwise break the rules of the real world. Of course, if you start to interpret a standard text in this creative way, you can justify anything. A reasonable interpretation must assume some general properties of the application domain it applies to (and in C there also more specific rules about how requirements are to be interpreted in 5.1) and must assume that wording such as "for which" has actual meaning that needs to be taken into account and can not simply be ignored. In C23 "Note 3" was added that aims to clarify that such nonsensical interpretations are not intended:
"Note 3 to entry: Any other behavior during execution of a program is only affected as a direct consequence of the concrete behavior that occurs when encountering the erroneous or non-portable program construct or data. In particular, all observable behavior (5.1.2.4) appears as specified in this document when it happens before an operation with undefined behavior in the execution of the program"
Ie:
int ub(int idx, _Atomic int* x)) {
char y[1000];
int r = *x; // 2
r+= y[idx]; // 1
return r;
}
This opens a big can of worms.
edit: from a few quick tests GCC seems quite good at preserving ordering of potentially faulting instructions across side effects, even when reordering would be profitable (for example hoisting operations out of loops). It might be a posix requirement in practice because signals make them "visible".
edit2:
ok, this "fails":
extern volatile int x;
int ub(int d, int c) {
int r;
for (int i = 0; i < 100; ++i) {
r+= x; // 1
r+= d /c; // 2
}
return r;
}
Potentially this could be fixed by unrolling out the first iteration of the loop and preserving the first volatile access above the division, which can then be cached. This would also help with the variant where the volatile access is replaced by a function call that currently gcc doesn't optimize.
The reordering of a volatile access after a potentially trapping operation is not conforming. I think it is an important property of volatile that it prevents this optimization, so I hope that GCC will be fixed eventually. A potentially trapping operation can also not be hoisted above a function call, and compilers that did this all got fixed in the mean time.
https://developercommunity.visualstudio.com/t/Invalid-optimi...
There’s something quite arrogant about how’s it’s configured which I can’t quite put into words
Are there any reasons why that is so? Do compilers not reuse the information they gather during compilation for diagnostics? Or is it a deliberate decision?
I'd imagine the generic case becomes a non-trivial problem if you don't want to produce fluke/useless diagnostic messages.
The compiler might already be several optimization passes in at this point, variables long since replaced by chained SSA registers, when it suddenly discovers that an IR instructions produces UD. This itself might end up being eliminated in a subsequent pass or entirely depend on a condition you can't statically determine. In the general case, by the point you definitely know, there might not be enough information left to reasonably map this back to a specific point in the input code, or produce useful output why the problem happens here.
Back in the day when RAM wasn't so cheap that you could find it in the bottom of a Rice Krispies box, I worked with a C++ codebase that required us to find a dedicated compilation machine because a standard developer loadout didn't have enough RAM to hold one of the compilation units in memory. Many of these tools (gcc in particular, given its pedigree) date back to an era where that kind of optimization mattered and choosing between more eloquent error messages or the maximum scope of program you could practically write was a real choice.
They do try to run things so everything's used. They also try to compile quickly. There is a conflict.
GCC does, but I guess this case just a case of missed warning, possibly to suppress false positive cases.
void foo(bool cond) {
int a = 0;
if (cond) a = 10;
if (cond) printf("%d\n", 10 / a);
}
void foo(bool cond) {
int a = 0;
if (cond) {
a = 10;
if (cond) printf("%d\n", 10 / a);
} else {
if (cond) printf("%d\n", 10 / a);
}
}
Now, maybe there are some passes across which false-positives would not be introduced, but that's a rather small subset, and you'd have to reorder the passes such that you run all of them before any "destructive" ones, potentially resulting in having to duplicate passes to restore previous optimization behavior, at which point it's not really "reusing".
[1]: https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.h... [2]: https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html
I'd actually go the other way: I think most practicing programmers have not read the entire standard specifying most languages they use day-to-day and have no real idea what the abstraction-break looks like that turns their code into a format consumable by the next layer down.
How many Java programmers do we assume know anything about the bytecode of the JVM, for example?
The corresponding Java, C#, Python standards, equally with the standard library, are even bigger.
To come back to the point, many don't know how deceptively complex Python happens to be, even though on the surface looks like a BASIC replacement.
If we compare to equivalent Python code, for example, the behavior is simple and straightforward: division by zero causes a runtime error that is reported in a well-defined manner. Similarly with other arithmetic issues in C++ that can trigger UB - e.g. integer overflow is just not a thing in Python (short of OOM). The detailed rules may well be complicated, but it doesn't matter as much when the behavior is intuitive and conforms to common sense expectations.
Indeed, it isn't. The check becomes useless if it happens after the division, though.
No, they cannot. You don't have the right to make any assumption about integer overflow in C.
Most of the whining about UB is from people who still refuse to accept that you don't have the right to think about your processor family once you write in any language that is not assembly.
Since when is it reasonable to assume that?
> Then you have weird edge cases like assigning the return value of a two argument std::max involving temporaries to a reference
You have a reference to a temporary. Reference lifetime extension is a thing. No UB there. Completely defined and supported.
However, some benchmarks use iteration on a signed integer, and assuming that loop terminates makes it slightly faster, so in order to retain that marginal advantage over other languages, signed iteration shall be assumed to never overflow.
This is very typical of the C++ experience.
As a slightly contrived example, assert(sizeof(char) == 1) is true by definition and could be elided, but it might be a useful reminder to see it in the source, next to code that implicitly relies on this truth.
I don't think any programmer puts UB on purpose into the code, even if they can enumerate from memory all 200 or so cases of UB just in the C standard (no idea how big the list is in the C++ standard - thousands maybe?).
The original sin was compiler writers exploiting UB for optimisations in bizarre ways instead of working with the C and C++ committees to fix the language to enable those types of optimizations without requiring UB, or at least to classify UB into different 'hazard categories' (most types of UB in the standard are completely irrelevant for code generation and optimization)
I agree with you. But the problem is not people putting UB in their code, either on purpose or by mistake: we all do that, every day!
The problem is people trying to defend their code once it has been made clear to them that it contains UB, and trying to fight the compiler rather than fix their error.
How about when code is written correctly and then later the standards body makes previously implementation defined behaviour into undefined? I've got some code that calls realloc that WG14 declared to be undefined years after I wrote it and I doubt that experience is unique.
The evangelical attitude that the standard committee knows best and your code is wrong and you should immediately down tools to work around the compiler noticing the opportunity to miscompile it is quite popular. I think it's a really compelling argument to build nothing whatsoever on the ISO C/C++ stack and replace what you do have with something less hostile, aka anything whatsoever - rust, python, raw machine code written in hex - none of them have this active hostility towards the dev baked into the language design.
For realloc, different implementations did different things and clearly said they will not change. There wasn't really any other choice. If your program was written for one implementation where it works, it can continue to do so, but it was never portable to other implementations. The standard now simply reflects this reality.
Thank you for pointing this out.
Put the two together and you get a fast and fragile language implementation. I know why the benchmark people push the compiler in that direction. I'm doubtful that WG21 or WG14 especially want this emergent property.
My suspicion is that this is an accident of history that has too much unwarranted inertia behind it. The moral stance that it's all lesser programmers erroneously writing wrong code is aggravating in that context as it actively opposes anyone making things better.
Compare to the concurrent memory model: while DRF-SC still has a plenty of UB, it is at least possible for a competent programmer to figure out the correctness of their code.
I certainly do not believe that is realistic to stamp out all UB from C and C++ (at least while pretending that the resulting languages have anything to do with the original ones), but there is a lot that the standard could do to try to limit the most egregious cases, possibly providing different levels of conformance (like it is done for floats and IEE754).
[1] of course implementors are part of the committee so they are not blameless.
I don't know how constraining is this more restrictive implementation of the standard is, but certainly it will help with maintaining a bit of sanity.
Further, those 'checks' might be in place for a long time, outlasting compiler versions and perhaps even language standards.
That came out wrong. What I mean is that it's not a threat from the standard that its buddy the compiler will ruin your computer, your day, and your life. Instead, it's a promise from the programmer to the compiler that the programmer won't perform some operations.
Since the programmer promised not to do some things, the compiler assumes those things aren't done while reasoning about the code, but if it turns out one of those things was done, the compiler's whole chain of reasoning is potentially invalid. So the compiler's options are to trust the programmer, and make the assumptions, or to instead treat the programmer as a dirty liar, and not make any of the assumptions while expending much effort considering all the subtle ways the programmer could try to trick it. Actually, there's a third option, trust but verify, where the compiler trusts the programmer to make a best effort, but verifies the assumptions when convenient. Unfortunately, it's not often convenient to verify in C or C++, but broadly compilers are getting better at it over time.
I'll just add that the third option is usually not considered viable because the main point of using C/C++ is for performance, thus people writing code want their compiler to optimize it as much as possible.
It may be possible to limit the bad effects of faulty assumptions by being very careful about how deductions propogate from unreliable assumptions. However it's certainly not easy, especially without throwing out all optimizations that interact with a potentially undefined operation.
It is possible to design a languge such that a compiler doesn't need to make assumptions to perform optimizations, if programs are required to give enough information to compilers that they can statically prove properties they need to perform optimizations. But C and C++ are not designed that way, and it can't easily be retrofitted on top of them.
I ask because when I wrote that post, I couldn't remember if I had heard the term before. In other words, I am not sure if I copied someone else or not, and if I did, I want to add attribution.
(NB entirely. That is, it's reasonable to coalesce arithmetic expressions together or inline functions such that a single operation in the output has subsumed multiple lines of code. However, if for a source S you can remove one or more entire lines of code to produce source S' with the same output, then something is wrong with the intent of the program or the compiler has misinterpreted it)
I suppose some of this depends on what you mean by "eliminating a line of code", as an optimizing compiler pretty quickly transforms the program into a state where any kind of 1:1 mapping between lines of code and instructions is impossible.
I don't think hoisting would be covered by this suggestion, pjc50 suggested deletion rather than moving.
I read the proposal very literally, which also means I'm unclear what options you have in your final paragraph:
Compile function as is; For each line, make a copy with that line missing and recompile; when this identical to the original, flag that line as "warning: optimised out of existence"
Optimizing compilers can also produce different output but with identical semantics depending on a lot of heuristics. What happens when deleting a line of code produces different inlining or specialization decisions? What is a rigorous definition of "identical"?
"Deleting a line" is also not the only way that unexpected and perhaps unwanted behavior from UB reasoning appears. Unexpected code motion is absolutely a thing that people have complained about. I don't think you can so quickly insist that things like hoisting are out of scope here.
As a compiler--indeed, optimization--developer, one of the things that continually frustrates me about discussions on UB is that so many suggestions are based on just complete incomprehension of how compilers work. One of the first things compilers do is throw away the original source code; outside of initial front-end codegen, there is no reference to the original AST, and instead everything is based on a pseudo-assembly IR. A concept like an "entire line of code" just doesn't exist in IR; even mapping instructions back to line of codes is a difficult process because frequently they aren't even attributable to a line.
Enough is retained to allow line-step debugging. Godbolt will even do a nice mapping for you between source and object lines. Yes, a lot of it gets thrown away. My point is "this line of code has no effect on the output" is a problem, and the developer should be warned rather than lulled into a false sense of security that their check is doing something.
(Code coverage has the same issue of doing line attribution, plus the difficulty of hitting error conditions in practice, but it would also show unexecuted lines of code)
Only without optimization. Every compiler I have ever used (and I have used a lot of them!) starts having difficulty with step-debugging at the lowest-level of optimization, typically -O1. Some functions work, most don't.
That doesn't have to be an immutable fact for all implementations of compilers for all languages. The Rust compiler has multiple IRs (AST, HIR, THIR, MIR & LLVM IR), and for all but the last one (codegen), we keep around, if not a reference to the HIR node that allows us to navigate its context, at least a Span that points at the user's code (sometimes a whole struct or linked list of "facts" we've collected so far, each containing Spans). rustc also throws away information to reduce memory consumption, much to my dismay, but we still perform later checks to re evaluate (sometimes as an approximation) what the reason for a certain "decision" was, so that we can talk to the user in terms they understand, and not in terms of compiler internals.
The ideal design would be to have a single implementation of all logic with two modes of operation: the low memory consumption one (what you state as the way compilers work), and a "track everything without discarding metadata until the end" mode, that can provide better human readable output.
(For instance, in QEMU we have code like: "if (kvm_enabled()) { do_something_kvm_specific(); }". In a no-KVM build, kvm_enabled() evaluates to compile-time false, and in a KVM build it does a runtime check of whether the user turned on KVM for this run.)
The point is that there is still value in that - your running program can at least say "oops" after the fact. You'll at least get errors when running your tests!
Removing the `printf ("oops")` altogether under the guise of "well, something 3 lines above was UB, so we can delete the rest of the program" is what users are complaining about.
The way the compilers go about it now, is that they don't even let you fail tests because you made an oopsie a few lines ago and now everything after that is considered safe for deletion.
Look at this example:
char *myvar = null;
...
myvar[0] = 'A';
...
if (myvar == null) {
printf ("oops\n");
}
1. I don't want the 'myvar[0]' assignment removed because the compiler determined that mvar is NULL. If it's there, at least I'll get a crash before the 'oops'. If the crash doesn't come, at least I have the 'oops' to warn in testing.
2. I don't want the `myvar == null` check removed. If the crash in the assignment didn't happen, I **want** the message 'oops' printed.
The compiler is clearly going against the intention of the programmer if it fails to emit code for any of those lines, because it turns a failing test into a passing one.
If you want to be able to observe values through invalid pointers then the compiler can't do things like store locals in registers since maybe you care about checking a value through one of these pointers. Huge mess! Like, whose to say that the line "myvar[0] = 'A';" doesn't write to the location in memory where the constant string "oops\n" is stored? How will the compiler ensure that your program does what you expect after a bogus write somewhere in memory?
I'm not asking for that. I'm asking that the compiler not remove code because of a prior error!"
If the program prints garbage, that's good enough to fail on a test.
If the program cannot print anything at all, then so be it.
What's actually* happening is that an over-eager compiler is removing something that exists. I'm asking that it not do that.
I'm not asking that it ensures that the line after the error executes, I'm asking that it not be eliminated.
I'm not sure what 'eliminated' even mean here. If the pointed to object has been replaced by a register, there might not even be a pointer to check. I.e. there might not be any meaningful code the compiler could generate for the line.
Compiler transformations attempt to preserve semantics of the code, do not attempt a faithful line-by-line translation (beyond toy compilers). If there is no semantic for a line, how can the compiler translate it?
Consider this code:
static inline int * get_x(void* ctx) { return (int*)ctx; }
static inline int indirect( int*(*getter)(void*), void*ctx) {
int * ptr = getter(ctx);
if(!ptr) { return 0; } // 1
return *ptr;
}
int main() {
int x = 10;
return indirect(get_x, &x);
}
It doesn't even make sense to warn, as main, get_x and indirect could be in completely independent libraries and each make sense as written on its own (getter+ctx is just an hand built closure, so the code is far from non-realistic).
I dunno if that is relevant. In the example you posted, simply removing the `static inline` leaves a pointer to check. The value isn't moved to a register.
We are arguing the cases when the source lines are there, but then are not emitted. In the code samples under discussion, the pointer values are still there and can be checked. Their values are not sitting in some register.
> (getter+ctx is just an hand built closure, so the code is far from non-realistic).
Qualifying with `static inline` is unusual, and none of the objections are using static+inlined code as samples of poor code generation. The examples being presented are a lot more real (as they are taken from existing project, not contrived to make an argument).
The code under discussion does not compile, so it is hard to discuss it. Please consider this variant: https://godbolt.org/z/EMzr4naMP
As you can see, while the check is still present in the non-inlined foo, main doesn't actually call it and omits the check. There is nothing left in main to check as there is no myvar object left.
This example and my previous one are similar: in both examples, not only the compiler could statically compute the value of the pointer, but it can track the pointee directly ('int x' in my example, the null pointer in your), hence it doesn't need to allocate any memory or registers for the pointer.
Separately, in both cases the compiler was able to statically prove that the pointer must be pointing to something (in my example because it knows the target, in yours because of the assignment through it), so the null check can be removed as the value of the condition is statically known to be always true.
Finally, I believe that in your example GCC found the contradiction with the pointer being both null and not null and realized that the whole main function is not possibly reachable, hence it isolates it with ud2 (clang is just silly).
What code would you expect GCC to generate in main to test the condition? Should it allocate a register, initialize it just for the purpose of performing the branch? Should it do it just for your example or also for mine?
https://www.youtube.com/watch?v=w3_e9vZj7D8
Compilers generaly do not abuse UB (outside of compiler bugs), its just that UB is a very missunderstood subject.
I watched the video and didn't learn anything new. I know UB pretty well.
> Compilers generaly do not abuse UB
It's disheartening to see this take from the guy who said that one word broke C.
It is also false, in my experience because some people, including compiler developers, think that compiler freedom is the definition of UB, not a side effect of that one-word change:
https://gavinhoward.com/2023/08/the-scourge-of-00ub/
So your claim in the video that such things are scary to compiler developers is sad. They have broken existing code; they are not afraid to.
I used to be somewhat confident (https://gavinhoward.com/2024/05/a-grateful-open-letter-to-je...) in the direction of C with you and JeanHeyd Meneide on WG14. But if you sincerely believe what you just said, then it seems you have changed your mind since you wrote that one word changed C. So I am not as confident anymore.
00UB does need fixing, and C needs less UB. If you don't think so anymore, then C will become untenable for me to use.
> I understand C way better than i used to.
Maybe, but I did not see it in that video.
What I do know is that you do not know much about the humans using C or the environments where C is used. C is a tool for humans, not machines. Prioritizing machines over humans will make C worse over time, and implementors will use that to further excuse their behavior.
Perhaps it is. It wasn't always so.
In K&R the term "behavior is undefined" occurs often. Everyone understood it's meaning. It meant "you get what the hardware gives you", meaning the compiler will output the same instructions it always does, but K&R didn't say what those instructions would do (typically because it couldn't). Of course what the hardware did on any given arch was perfectly well defined.
The definition had it's upsides and downsides. On the upside, programmers took advantage of their knowledge of the hardware they were targeting to write efficient code. Embedded programmers tend to do that sort of thing fairly aggressively (for example, there is often something useful stored in location 0). The downside is if they did that then their code wasn't portable.
The definition gets ugly if the programmer is trying to write portable code, because it means they don't get warned if they code they wrote wouldn't port easily. As a consequence writing non-portable code was and remains easy mistake to make. The sane solution was a --error-if-not-portable compiler option.
But that's not what we go is it? Instead the meaning of "the behavior is undefined" morphed from "hardware defined" to "implementations are allowed to assume that the respective runtime condition does not ever occur". From what I can tell compiler writes turned that definition into "the behavior is compiler writer defined" so they could gain some edge in the "who has the best optimiser" games they love to play. Consequently the definition the compiler writer uses is almost always "delete the code".
But doing that made it harder to write correct code. Whereas before the code always had the same meaning on some hardware, it now changes it's meaning depending on the same hardware on whether you supply -O0 or -O2. And it does so without warning, because we never got the --error-if-not-portable option. The result has been numerous bugs. For example take:
int parse_packet()
{
uint8_t buffer[1500];
int len = read_from_internet(buffer, sizeof(buffer));
uint32_t field_size = ntohs(*(uint32_t*)buffer);
if (buffer + field_size >= &buffer[len] || buffer + field_size < buffer)
return -1; /* error return */
/* continue parsing the packet. */
return 0; /* success */
}
There were two reasonable outcomes for this code. One is the K&R approach. The other was the --error-if-not-portable approach which means refuse to compile the code. I think most compiler users (as opposed to people playing word games in order to win some optimisation game) would call what actually happened "compilers abusing UB". That because no one wins from that particular "optimisation", except the compiler writer doing some micro benchmark. At best the programmer had a flaw the compiler knew about and exploited, but didn't warn him about. The users of the compilers output got hit with a CVE.
That's the best interpretation. The worst is the C standard committee has lost the plot. Their goal should be to produced a simple, clear standard even a novice programmer could safely pick up and read to learn the language. That is what K&R was. Instead we've arrived at the point governments are saying the language is too dangerous to use.
Serious logic error, surely. "Can never happen" does not follow.
Sure, looking at the assembly it looks like the compiler has removed the 'if' branch, but, under "as if", it's equivalent to the compiler having rewritten 'a/b' to 'if (b==0) { printf("B\n"); } a/b', which strictly only modifies behavior at the UB.
And, while it may seem strange that UB results in such "copying" of code with minor alterations, that's not particularly rare at all. Take '(uint32_t)100 / someU32' on RISC-V, which'll result in -(uint64_t)1 on someU32==0 via the simple implementation of divu or divuw, despite that not being a valid uint32_t value, which'll result in significantly weird following behavior despite just being the most literal lowering.
this is subtile misleading unspecified and undefined behavior are not quite the same
UB means it's (simplified) specified that the compiler is allowed to do whatever it wants (or more concretely in most cases is allowed to assume that a specific thing is impossible when optimizing to a point where of it does happen pretty much anything can happen including things like it seeming that an int is two different values at the same time or (which might still be one of the more harmless wtfs which can happen))
But please change the standard to disallow 00UB:
https://gavinhoward.com/2023/08/the-scourge-of-00ub/
I have to do a lot of stuff to avoid UB in my code. I even implemented UB-free two's complement arithmetic using unsigned types just to avoid all of the problems.
And it has definitely gone too far when people think that the compiler doing anything on UB is the definition of UB, not just a side effect.
"Note 3 to entry: Any other behavior during execution of a program is only affected as a direct consequence of the concrete behavior that occurs when encountering the erroneous or non-portable program construct or data. In particular, all observable behavior (5.1.2.4) appears as specified in this document when it happens before an operation with undefined behavior in the execution of the program."
^1 https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3220.pdf
However, I think that consistency should win here. Given that unsigned arithmetic already wraps around - and this cannot be changed for backwards compatibility reasons - making signed arithmetic do something different is just bad design. It would certainly be better if the default behavior was to trap rather than wraparound or saturate, but, well, we're talking here about a language that is >50 years old, so that ship has sailed.
It would sure be nice to get proper dedicated operators for trapping and saturating arithmetic in some future version of the C standard, though.
C is low-level enough that it's not particularly useful to think in those terms, IMO. If you're writing in C, it's usually for one of the two reasons: either you want to be "close to the metal" because you're doing low-level stuff, or it's a legacy codebase. From the first perspective, I can't think of any "modern" - as in, past three decades - architecture for which signed wraparound is not the default & fastest behavior. From the second perpective, a lot of legacy code actually assumes signed wraparound (because compilers used to just defer to what the hardware actually did).
FWIW the reason why C had UB for signed overflow historically is because back when it was being standardized, there were still machines around that used something other than two's complement to represent signed values. The original ANSI C89 straight up refused to specify the signed representation, and C99 narrowed it down to several options; thus, the obvious hardware implementation would in fact produce different results on overflow. Conversely, unsigned integers always had a well-defined representation for which wraparound was simply the natural hardware behavior. I don't think this was ever meant to have some kind of higher meaning.
Given that C23 explicitly mandates two's complement for signed now, IMO, the signed overflow behavior should just be made consistent with that as well as real-world hardware. The only claimed downside to this is that compilers can no longer "optimize" code that was previously broken but would now be well-defined, but I don't see why that is an actual problem.
I am programming C because Python is too slow for my use case (and too annoying and to unstable) and yes, I use C's integer types as an abstraction for mathematical integers. And I would say this is the case in almost all C code I see and most programmers I talk to. How would two-complement's wrapping for int be useful for anything? It does not model anything useful IMHO.
And the problem with compilers not optimizing is that people want them to optimize... Compilers not being able to to do this as well anymore would indeed be an actual problem for many people.
if (x + 1 < x) // overflow
I don't disagree that people want compilers to optimize in general. But can you give an example of optimization that is 1) relies on compiler treating signed overflow as UB, and 2) is actually useful (i.e. makes conforming code run faster)?
As far as modeling, the very fact that C ints are bounded - and lest we forget, unless you use `long long` everywhere, even `long` doesn't guarantee you more than 32 bits per the Standard, which is not all that large! - they are not a good abstraction for mathematical integers. C developers use it as such in practice because it is convenient, but consequently tend to ignore the overflow behavior, not the least because dealing with signed overflow is so inconvenient. Even today, I suspect that the vast majority of production C code out there is not actually safe wrt to signed overflow-induced UB due to overly large inputs (e.g. how many CLI tools do such validation for values that come out of argv?).
Your definition of a far more complicated dance is absurd.
Signed integer being undefined makes a lot of integer math faster actually: https://kristerw.blogspot.com/2016/02/how-undefined-signed-o...
There's a reason zig made unsigned overflow undefined: it allows for more optimisations (Zig can also afford it thanks to their wrapping addition (+%) operator).
With C23 there are also now checked integers to do overflow checking for arithmetic operations.
Nothing on you, but in my mind, this is a sign of just how brain dead and unprofessional our industry is.
Using hardware integers as mathematical integers is like using aluminum instead of titanium for a high-stress part in a fighter aircraft. Sure, it does just fine in normal flight; what's the big deal?
And then the pilot needs to pull a 10-g maneuver, putting 10 times the stress on that part. And then the pilot dies because the aircraft breaks apart in midair.
If you need mathematical integers, use GMP. Or implement your own. It isn't hard. I implemented rationals using 32 bytes and no allocations when integers are no bigger than hardware integers.
Adding all of that with this:
> You realize that I am part of the standard's committee? ;-)
makes me geuinely fear for the direction of C.
I think I may need to write boringcc to rescue C from itself. https://groups.google.com/forum/m/#!msg/boring-crypto/48qa1k...
Big number integers also has use cases, but this is has very special purposes. And also big numbers are not "true" mathematical numbers if you want to be really pedantic, because if you memory is limited also this abstraction breaks downs at some point. Models for true mathematical integers do not exist in the physical world, so that "int" isn't one, is meaningless pedantry.
Both of those should use unsigned integers, and they are limited by the size of the machine, so no, they don't need a mathematical model of integers. A good machine will have a size_t size that can hold any size of object, and by extension, any number of elements of any size, including char.
So for counting things and loop indices, size_t should be used, not any signed types.
In my code, I essentially use just size_t and unsigned char (because the standard did not specify the signedness of char up to at least C11). If I use something else, I am checking bounds.
> Most people could not calculate with higher number in their head, are you saying their mental model of integers is a small integer model different from mathematical model?
Are you seriously trying to accuse me of this? I am the one saying that hardware integers are not a sufficient abstraction for mathematical integers, yet you say that I assume that people have a small integer mental model?
No, I am not. I am the one telling you that using hardware integers in place of big ints is not good.
> Models for true mathematical integers do not exist in the physical world, so that "int" isn't one, is meaningless pedantry.
True, every abstraction is leaky, but big integers and rationals can get so large that it doesn't matter.
And yes, big ints are a leaky abstraction, but they leak so much less than hardware integers because they don't wrap and because they are not subject to 00UB when unsigned types are used.
You are correct, but I care about correctness more. I will use a lobrary to get it.
That said, having such checks by default would surely reduce the possible impact of overflow on security, so I would support it.
Making it UB in ISO C ensures that no portable program can rely on a specific behavior and this is what makes it possible to find bugs this way, because it is plausible to assume that a program with overflow is buggy. This is also why this does not work for unsigned. We can easily change a compiler to trap on unsigned wraparound. But this is mostly useless because there would be far too many false positives.
This is true.
> and this is what makes it possible to find bugs this way
No, this is not true. In fact, it is the opposite of true.
If a program cannot rely on behavior, it cannot rely on having the behavior that causes the bug to be visible.
For this reason, signed overflow is essentially a solved problem, while unsigned wraparound will be the source for many interesting bugs - exactly because it is not UB!
Does the compiler faithfully compile the source code with sanitizers? Of course not, so we still cannot rely on them.
> So you can not distinguish between intended wraparound and incorrect wraparound.
This is exactly why I implemented both types in my arithmetic. The one I used is the one I intended.
int i = 0;
// nullptr dereference
int ub = *(int*)i;
"A pointer whose value is null does not point to an object or a function (the behavior of dereferencing a null pointer is undefined), and compares equal to all pointers of the same type whose value is also null."
if (((void*)0) != (void*)(0, 0))
puts("true");
https://en.cppreference.com/w/cpp/language/nullptr
https://en.cppreference.com/w/cpp/language/pointer Nullptr section is not directly linkable, but:
"A pointer whose value is null does not point to an object or a function (the behavior of dereferencing a null pointer is undefined), and compares equal to all pointers of the same type whose value is also null."
If yes, then the spec says regardless of how the implementation defines nullptr, assigning literal 0 to a pointer variable is assigning a nullptr (i.e. the compiler must turn `0` into whatever the nullptr is in the underlying implementation).
If no, then 0 is just some unknown value and the assignment gives you an implementation-dependent (not necessarily invalid) pointer to whatever address `0` means on this architecture.
(Note also that in C, unlike C++, "const" really means that it is read-only, not that it is a constant expression, so that doesn't change anything.)
However, even if the resulting pointer is not a null pointer, dereferencing it is still UB unless it points at a valid C object. So if the compiler can statically prove that for this architecture, there's no valid C object at that address (e.g. because the compiler itself never produces code that would result in objects placed at that address), it can still treat it as UB, same as dereferencing a literal 0.
In practice, of course, a useful C++ compiler would let you do it, somehow. But that is firmly in the realm of implementation-specific stuff (keeping in mind that UB can mean anything including being well-defined by a specific implementation.
the dereference itself is UBI guess I was nitpicking. For gcc and the target architecture that is compiling that code it is UB so it is optimizing things with that information
Does "A pointer whose value null" always means that it is literally value 0? That is how it is in practice but how C++ handles architectures where 0 is a valid address that a program can dereference? Does cpp compilers use another value for nullptr in such architectures?
However, because it is UB, the compiler doesn't need to do anything special if there's no trap on null dereference, since "reading something" is a perfectly valid subset of UB. The only thing that it needs to ensure is that no global, local, or heap-allocated C object can end up at address 0, which is easily done by "wasting" a single word at that address (or using it for some internal purposes).
It may still be that there is something useful at address 0 that some code may want to access. For example, in real-mode x86, the interrupt vector table starts at that address, so if you want to change what gets called to handle INT 0h, you'd need to write into it. But at this point you're already deep into architecture-specific stuff, and so it's not unreasonable to expect you to use some special facilities that a compiler for this architecture might provide to enable it, instead of the usual pointer dereference.
The C spec only requires that integer constant expressions that have zero value are nullptrs. The moment you assign it to an int variable, it's no longer a constant expression as far as the language is concerned, so if you then cast that int variable to a pointer, the result is indeterminate
An implementation could declare that there is a valid object at address 0, I suppose, but strictly conforming code cannot rely on this. As far as I remember, the only ints you can cast to a pointer and then safely use are those that were cast from (valid) pointers in the first place.
For every C and C++ standards compliant compiler it is UB. Every single one. By definition, whether or not the compiler then takes advantage of that. It may or may not be legal on the target machine in question, but C and C++ don't target the machine in question, they target an abstract machine with certain properties. The behavior of the abstract machine is then translated to behavior on the target machine. Dereferencing the nullptr on the abstract machine can result in any behavior at all, including what you describe here, but it is not promised by the standard.
If you want to access memory at address zero, you cannot do it in a defined way from within the language. You can drop to assembly or use compiler-defined extensions, but if you don't do it like that, you are playing with UB.
Developers must avoid UB in their code, at all costs. Wondering what a specific compiler will do with your UB code is useless: as soon as you realise you have UB, go and fix it!
If a compiler is certain there's UB should it format the developer's hard drive immediately, or does it have to wait until the program is run before formatting to be standards compliant?
On a more serious level, should it emit a warning? If it knows UB is reached, a warning sounds reasonable, but it's pretty rare that a compiler is 100% certain something is reached, and that analysis is probably not worth the cost.
The standard has enough variants of UB that both options are possible!
For normal runtime UB (e.g. division by zero), it needs to wait until it is certain that the undefined code will actually be executed. So compilation/linking must not fail, the behavior is only undefined if the program is actually run.
IFNDR (ill-formed, no diagnostic required) is a different kind of UB; here it is expected that compilation/linking may already fail.
Then there's also the infamous "preprocessor UB" (e.g. https://wg21.link/p2621), which reuses the "undefined behavior" terminology despite not having anything to do with runtime semantics. I guess this is where the compiler might format your hard drive (old gcc versions actually launched nethack on undefined pragmas!).
The resulting "reasoning" can be counter-intuitive, and seeing what happens and what problems it can introduce must be instructive to people who are not aware.
Better informed/educated C programmers is an improvement on the state of the world, in my opinion.
UB is a post-facto justification for compiler optimizations that only work on non-buggy programs. It's a circus because programs have bugs.
Given that reality, knowing what your compiler will do with UB is useful and important because you're going to see it all the time in debugging. Even if it's ugly, it needs to be treated as a familiar face.
The more you refuse to accept undefined behaviour, the more it will hurt when it will bite you!
Stop turning around the problem and fix your code.
Given that, plus the reality that we insist on continuing to write a million lines of C++, treating UB as something you'll see frequently in your career is probably the reality-based behavior.
Sure it is! I'm not talking about skills: everybody writes UB, at any time. I'm talking about attitude: when you see UB, you must strive to remove it, not to protect it!
I didn't recognize it as your position because you presented "Developers must avoid UB in their code, at all costs" as supporting the preceding statement "This post has no point;" if your point is "developers should remove UB when they see it," then the point of this post is helping people see what UB looks like at the next layer of abstraction down as an aid to identifying and removing it.
(The C++ standard is over 2,000 pages. It's impractical to believe developers see UB by reading the C++ source code; most is identified by an application behaving incorrectly followed by inspection of the machine code to understand where the developer's assumptions about the C++ specification failed to match with the actual specification).
Maybe I did not choose my words properly, I apologise.
You seem to care about developers avoiding UB. Exploring how compilers warn or don't warn for some of these cases is very meaningful - maybe it can lead to new warnings, or better tools so that developers can avoid UB.
I really don't understand why the C standards body can't just define what should be the intended failure behavior in case of specific UB cases, and then the compiler developers to onboard this spec. There should be no impact to backwards compat because
a. this will be a new C version b. No program ever should be defined around the UB behavior
But UB still exist in 2024 and will likely do till I am too old to whine on the internet about it.
Remember that the failure mode must be a) implementable, b) implementable at best minor loss of performance, c) not break the ABI.
I agree that C and C++ have way too much UB, but no UB is a pipe dream.
Implementation defined
> Or a write past the end of an array?
Ditto
> Remember that the failure mode must be a) implementable, b) implementable at best minor loss of performance, c) not break the ABI.
And "Implementation defined" matches that for the two examples you gave.
What it does is force the compiler vendor to say "If you do $THIS, we will do $THAT."
At least then users can make informed choices, sort of like "Clang guarantees that dereferencing a NULL pointer results in an effective address of zero, but GCC makes no guarantees, therefore we'll be going with Clang".
I mean, seriously, there's a lot of footguns in C only because the standards writers said "Undefined behaviour" and never expected that compiler authors will take that to mean "remove code, or anything else, including nasal demons".
Well, yes.
"Writing to memory that is not available or allocated will always cause the instructions for the write to be emitted. The statement, expression or function call that performs the write will not be eliminated."
The definition above is better than UB, which boils down to "Writing to memory that is not available or allocated may or may not cause all past and future lines of code to be eliminated from the emitting stage of compilation."
See the difference? The implementation-defined definition forces the compiler documentation to be honest: "We'll remove lines of code from your input" is better than "You broke the standards rules, so we get to do anything."
That would be a significant constraint on the optimiser.
Maybe. Maybe you work on GCC (or on Clang) and know the dirty details of the code inside ... I don't.
What I do know is that the performance in past C compilers which emitted every memory access unconditionally was considered "blazing fast".
If this is a blow to performance, how significant is it? Can you compile two benchmarks with and without the 'perform every memory access unconditionally' flag? Is there a flag for this? If no one ever did this, how can we tell there would be an impact on performance, nevermind significant impact?
Because I recall seeing benchmarks for things like eliminating `if (x+100 < x)` showing the performance impact to be not even a rounding error. I also recall seeing a benchmark somewhere for with and without the flag that removes null-checks, and once again that performance impact was, for all practical purposes, zero.
In the era of speculative pipelined execution, in respect of a test/check that would be speculated in the pipeline well before it is needed, how exactly will the code run faster if the test/check is removed from the sources?
Once again, I admit that I am not in the weeds in GCC or LLVM or CLang development, but I fail to see how eliminating an instruction that never slows the processor can have a noticeable impact on performance.
-O0 is close to what you want. For hand optimized code there isn't a huge difference between the various optimization levels, but for higher level code C++ code, the difference between -O0 and -O3 are sometimes one or two orders of magnitude.
And yes, not all, or even most, optimizations apply to all programs. All loop optimizations are really most relevant for numerical, vectorizable code; branchy pointer-chasing code won't benefit from them. Branchy code can benefit from other optimizations though that can bring their own issues.
Then, what does it mean for it to be "emitted", and how does it apply to e.g. C++ interpreters?
* Undefined behaviour lets compiler developers improve the performance of certain standard benchmarks (slightly but significantly) and they don't want to give that up.
* Most of the people who are interested in programming language design have already deserted C/C++ so the C standards bodies are dominated by the remaining traditionalists.
Because the effect wasn't always this bad; past C compilers did not look at a null-dereference and say "This means that we can simply skip the dereference". They emitted the faulty code which would then crash reliably.
Same with things like writing past the end of an array: you'd overwrite whatever was in the memory or stack. Now, if the compiler detects UB, it would silent discard it.
It's the whole "compiler silently doing something unexpected, other than exhibiting the wrong behaviour that the programmer expected." that makes the issue worse now than it's been in the past.