ISO C became unusable for operating systems development
arxiv.org
arxiv.org
I replied by pointing out that nearly every non-trivial C program would have to be recoded to work on that architecture. So what purpose did the Standard allowing that actually achieve?
I also see no problem for a vendor of a C compiler for that architecture making a reasonable dialect of C for it. After all, to accommodate the memory architecture of the x86, nearly all C compilers in the 80's adopted near/far pointers, and while not Standard compliant, it really didn't matter, and was tremendously successful.
D made some decisions early on that worked out very well:
1. 2's complement wraparound arithmetic
2. sizes of basic integer types are fixed at 1 byte for chars, 2 for shorts, 4 for integers, 8 for longs. This worked out very well
3. floating point is IEEE
4. char's are UTF-8 code units (*)
5. chars are unsigned
These 5 points make for tremendous simplicity gains for D programmers, and ironically increase portability of D code.
After reading the paper, I'm inclined to change the definition of UB in D to not mean it can be assumed to not happen and not be unintended.
(*) thanks for the correction
I believe the situation was that there were C implementations for DSPs (32-bit-addressable only) and IBM mainframes (36-bit addressable only), and when ANSI/ISO C was established, they naturally wanted their implementations to be able to conform to that new standard. So the standard was made flexible enough to accommodate such implementations.
Similarly why signed overflow is undefined behavior. There were existing implementations that trapped (CPU interrupt) on signed overflow.
I might have gotten the details wrong, but that's what I remember from reading comp.std.c (Usenet) in the 90s.
How many C programs today would work with EBCDIC? Zero? There's no modern point in C not requiring ASCII, at a minimum.
I'm all for extensibility mechanisms to shut down the people like this. You want EBCDIC? It's now a github repository; go ahead and maintain it, or shut up.
I'm guessing you mean that char is a UTF-8 code unit as you keep saying they're only one byte and a code point is far too large to fit in a byte / octet.
But that still seems very weird because a UTF-8 code unit is almost but not quite the same as a byte, so that users might be astounded when they can't put a byte into a char in this scheme (because it isn't a valid UTF-8 code unit) and yet almost no useful value is accrued by such a rule.
> a UTF-8 code unit is almost but not quite the same as a byte
D dodges that problem by having `byte` for signed 8 bits, and `ubyte` for unsigned 8 bits. That gets rid of the "is char unsigned or signed" confusion and bugs.
Still it seems weird to me to separate the concerns "This byte isn't a valid UTF-8 code unit" and "This string isn't valid UTF-8". I don't know what flavour of error handling D has, but I can't conceive of too many places where I want to handle those cases differently, so if they're separate I have to do the same work twice.
Also, one very natural response to invalid UTF-8 is to emit U+FFFD (the replacement character �) but of course that doesn't fit into a UTF-8 code unit, so an API which takes a ubyte and gives back a char must error.
As for invalid code points, you get to choose the behavior:
1. throw exception
2. use the replacement char
3. ignore it
All are valid in real life, and so D is accommodating.
It's unclear to me whether there's actually enforcement to ensure char can't become say 0xC0 (this byte can't occur in UTF-8 because it claims to be the prefix of a multi-byte sequence, yet it also implies that sequence should only be one byte long... which isn't multi-byte)
I spent a while reading the D website and came away if anything more confused on this topic.
Back in C infancy days, there had existed architectures where a byte could hold 9 bits that C compilers had to be written for. The 36-bit PDP-10 architecture springs to mind, and some Burroughs or Honeywell mainframes had those – I remember reading a little C reference book authored by Kernighan, Ritchie and somebody else explicitely calling out the fact that a C implementation could not rely on the fact of the byte always being 8 bits long and also stressing that the «sizeof» operator was reporting the number of bytes in a type irrespective of the bit width of the byte.
9 bit byte architectures have all but perished, however, C has carried the legacy of creative days of the computer architecture design along.
But that was over 40 years ago.
Time to move on.
What's the current definition of UB in D?
> Plenty of software works fine with different CHAR_BIT values,
Most won't. And good luck writing a diff program (for example) that works with CHAR_BIT == 32.
> although some poorly-written programs do need to be fixed.
Blaming all the problems on poorly-written programs does not work out well in real life. Good language design makes errors detectable and unlikely, rather than blaming the programmer.
Since C traditionally doesn’t have much of a testing culture I’m not sure you can trust any decent sized project here. I’d even be surprised if you could change the size of ‘int’. And moving off 2s complement is definitely out.
Linux and Rust agree u32 is the right name for a 32-bit unsigned integer type and u64 is the name for the 64-bit unsigned integer type, but Rust calls the signed 32-bit integer i32 while Linux names that s32 for example.
It would of course be very difficult for C itself to declare that they're now naming the 32-bit unsigned integer u32 - due to compatibility. But Rust would actually be able to adopt the Linux names (if it wanted to) without issue, because of the Edition system, simply say OK in the 2023 edition, we're aliasing i32 as s32, and it's done. All your old code still works, and raw identifiers even let any maniacs who named something important "s32" still access that from modern "Rust 2023" code.
1. they are slower to type
2. they just aren't pretty to look at
3. saying and hearing `int` is so much easier than `eye thirty-two`. Just imagine relying on a screen reader and listening to all those eye thirty two's
4. 5 minutes with the language, and you know what size `int` is, as there is no ambiguity. I haven't run across anyone confused by it in 20 years :-)
Don't use fixed width types by default as printing them or converting to string correctly will get ugly: printf("... %" PRId32 " ...");
Generally avoid the use of fixed-width 8-bit or 16-bit types in calculations or you might shoot yourself in the foot due to integer promotion rules.
Use fixed width types only for things that pass the boundary of a system, like in HW registers, serializing, database storage or network packets, but cast only at the end when converting to the type (optionally followed by bitwise manipulations) and cast immediately to other type when processing (after bitfield extraction and such things, of course).
> The paper makes many good points, but I think the author is throwing out the baby with the bathwater by concluding that we should entirely get rid of this kind of Undefined Behavior. The point of this blog post is to argue that we do need UB by showing that even some of the most basic optimizations that all compilers perform require this far-reaching notion of Undefined Behavior.
<snip>
> I honestly think trying to write a highly optimizing compiler based on a different interpretation of UB would be a worthwhile experiment. We sorely lack data on how big the performance gain of exploiting UB actually is. However, I strongly doubt that the result would even come close to the most widely used compilers today—and programmers that can accept such a big performance hit would probably not use C to begin with. Certainly, any proposal for requiring compilers to curtail their exploitation of UB must come with evidence that this would even be possible while keeping C a viable language for performance-sensitive code.
> To conclude, I fully agree with Yodaiken that C has a problem, and that reliably writing C has become incredibly hard since undefined behavior is so difficult to avoid. It is certainly worth reducing the amount of things that can cause UB in C, and developing practical tools to detect more advanced kinds of UB such as strict aliasing violations.
<snip>
> However, I do not think this problem can be solved with a platform-specific interpretation of UB. That would declare all but the most basic C compilers as non-compliant. We need to find some middle ground that actually permits compilers to meaningfully optimize the code, while also enabling programmers to actually write standards-compliant programs.
if (i++ < i)
mov ax, i
inc ax
cmp ax, i
jge else
cc: warning: UB line: 123 file: foo.c
void oh_the_humanity(int *ptr, int val) {
*ptr = val + 1;
}
* UB: ptr may be pointing a float variable. (It's not illegal to assign a float* to an int*, it's only UB when you actually dereference it with the wrong type.)
* UB: val + 1 may overflow.
* UB: potential data race on writing *ptr.
* UB: ptr may be a one-past-the-end-of-the-array pointer, which can be validly constructed, but may not be dereferenced.
* UB: ptr may be pointing to an object whose lifetime has expired.
* UB: ptr may be uninitialized.
* UB: val may be uninitialized.
As you can see, UB is intensely specific to the actual data values; it's not really possible to catch even a large fraction of UB statically without severe false positive rates.
% cat foo.c
#include <stdlib.h>
int
foo(int *bar)
{
int baz = *bar;
if (bar == NULL) exit(2);
return (baz);
}
% cc -O3 -Wall -Wextra -c foo.c
% objdump -dr foo.o
foo.o: file format Mach-O 64-bit x86-64
Disassembly of section __TEXT,__text:
0000000000000000 _foo:
0: 55 pushq %rbp
1: 48 89 e5 movq %rsp, %rbp
4: 8b 07 movl (%rdi), %eax
6: 5d popq %rbp
7: c3 retq
% cc -O0 -Wall -Wextra -c foo.c
% objdump -dr foo.o
foo.o: file format Mach-O 64-bit x86-64
Disassembly of section __TEXT,__text:
0000000000000000 _foo:
0: 55 pushq %rbp
1: 48 89 e5 movq %rsp, %rbp
4: 48 83 ec 10 subq $16, %rsp
8: 48 89 7d f8 movq %rdi, -8(%rbp)
c: 48 8b 45 f8 movq -8(%rbp), %rax
10: 8b 08 movl (%rax), %ecx
12: 89 4d f4 movl %ecx, -12(%rbp)
15: 48 83 7d f8 00 cmpq $0, -8(%rbp)
1a: 0f 85 0a 00 00 00 jne 10 <_foo+0x2a>
20: bf 02 00 00 00 movl $2, %edi
25: e8 00 00 00 00 callq 0 <_foo+0x2a>
0000000000000026: X86_64_RELOC_BRANCH _exit
2a: 8b 45 f4 movl -12(%rbp), %eax
2d: 48 83 c4 10 addq $16, %rsp
31: 5d popq %rbp
32: c3 retq
%
Fwiw GCC does have a related warning flag (-Wnull-dereference) but I'm not sure it's made exactly for this. I believe it works based on functions being annotated for possibly returning NULL, e.g. malloc. It's also not enabled by -Wall or -W because apparently there were too many false positives: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96554
I imagine patches would be welcome. I'm guessing there are more people who like to wish for more compiler features than there are people who like to develop compilers :)
https://godbolt.org/z/4TP1hfx4j
But your hint found -fno-delete-null-pointer-checks which does the trick
https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html
And -fno-delete-null-pointer-checks made it into LLVM too. It's a good to know but a little late for when I needed it, cheers :)
% cc -O3 -fno-delete-null-pointer-checks -Wall -Wextra -c foo.c
% objdump -dr foo.o
foo.o: file format Mach-O 64-bit x86-64
Disassembly of section __TEXT,__text:
0000000000000000 _foo:
0: 55 pushq %rbp
1: 48 89 e5 movq %rsp, %rbp
4: 48 85 ff testq %rdi, %rdi
7: 74 04 je 4 <_foo+0xd>
9: 8b 07 movl (%rdi), %eax
b: 5d popq %rbp
c: c3 retq
d: bf 02 00 00 00 movl $2, %edi
12: e8 00 00 00 00 callq 0 <_foo+0x17>
0000000000000013: X86_64_RELOC_BRANCH _exitLikewise, code generators shouldn’t be generating this faulty code.
Raising compiler error here is the only right thing to do.
There are of course more ambiguous examples, though obvious examples like the one above are sadly way too common.
* 1 and 4-7: Don't worry about where the pointer goes, as long as you treat it like an int in this function. If there are going to be warnings about improper pointers, they should be at the call sites.
* 2 If overflow will either trap, wrap, or behave like a bignum, then it's not the dangerous kind of UB, so no warning by default. Consider extending C so the coder can more easily control integer overflow.
* 3 Anything could race. Out of scope, don't worry about it.
See https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html
The problem is that except in a few trivial cases it is impossible to detect UB at compile time. Even whole program static analysis can only catch a small subset.
There actually is data, [1] and it goes against intuition because it shows that UB makes code slower.
It turns out that when programmers are faced with the possibility of UB, they (quite rightly) try to work around it. (In my case, I implemented signed two's-complement arithmetic entirely with unsigned types. I also implemented my own array types, including bounds checks when indexing the arrays.) These workarounds make their code slower, in general.
When you think about it that way, it makes sense because programmers almost never want UB in their software, so on average, the software that people care about will work around UB and become slower.
UB in C was defensible when platforms were not homogenous and when compiler writers did not abuse the spirit of it. But nowadays, most of it is indefensible because platforms are mostly homogenous, and compiler writers have been abusing UB for "performance."
We saw the same thing happen with speculative execution in chips. For years, chip manufacturers got away with lots of tricks to increase performance. Then Spectre and Meltdown were discovered. As a result, a lot of software had to be changed, which resulted in the software slowing down.
(Yes, there are now mitigations in chips, but I think those mitigations will be successfully attacked too. I think that it will continue that way until most or all speculative execution is removed.)
Likewise, with compilers exploiting UB against the original spirit of UB, which was just to make C easy to port to any architecture, we are probably going to have a reckoning about it in the same way we did Spectre and Meltdown. In a way, we already do, but it's spread out like a thousand paper cuts. Maybe that means that it will stay invisible enough that programmers never wake up; I hope not.
tl;dr: compilers exploiting UB actually slows down good code in much the same way that Spectre and Meltdown do.
[1]: https://www.complang.tuwien.ac.at/kps2015/proceedings/KPS_20...
THE primary UB of C is out-of-bounds write. Any proposal for UB-free C should address how to compile out-of-bounds write.
If you insist out-of-bounds write should compile in "portable assembler" way and reliably corrupt memory instead of allowing compilers to assume it doesn't happen, compilers can't allocate local variables in register. If it's on stack, out-of-bounds write may reliably corrupt stack, so how can it be on register?
"Portable assembler" people invariably say "that's not what I meant!" when compiler people raise this issue, but the question is "then what do you mean?" And I am yet to see any good answer.
I don't think anyone in my camp is insisting that the results be reliable in the sense you seem to take for granted. What we want is for the compiler to attempt to do what it's instructed to do, without eliding code based on counterfactual reasoning that assumes the absence of undefined behavior. In the absence of an explicit read that is being ignored, we're generally fine with cached values of local variables being out of date.
Maybe you could give an example that makes your point more clearly, and we can see if we actually disagree?
I thought about responding to jcranmer's post instead of yours, but wasn't sure how to approach it. It's a good comment, and I appreciated his attempt, but I feel like he's thoroughly missing the point of the complaints about UB. The complaint isn't that too much UB exists in C, but that compiler writers seem to use the presence of UB as an excuse for being allowed to break code that has worked historically. And the complaint isn't just that the code is broken, but that no apparent care is taken to avoid doing so despite the small amount of gain. It's a worldview mismatch, and I don't know how to bridge it.
Your comment seems about the same as the one I responded to. You seem to assume that people who complain about UB in C would have a problem with keeping local variables in registers, but I've never seen anyone actually make this complaint. Take for example the Arxiv paper we are discussing: he doesn't bring this up. This makes me suspect that your mental model of the people who are complaining about UB in C is probably flawed. I understand the technical issue you are gesturing toward, I just don't see it as being something that refutes any of the issues brought up in the Arxiv paper.
My hope was that a concrete example might help to clarify, but I do realize this might not be the right forum for that.
It's totally OK to say that an out-of-bounds write may or may not be detectable by reading any particular other local or global variable or heap value. That doesn't in turn translate to compilers being allowed to turn a bug in a SPECint kernel into an empty infinite loop (a case that happened!)
"Oh, it was quite a while ago. I kind of stopped when C came out. That was a big blow. We were making so much good progress on optimizations and transformations. We were getting rid of just one nice problem after another. When C came out, at one of the SIGPLAN compiler conferences, there was a debate between Steve Johnson from Bell Labs, who was supporting C, and one of our people, Bill Harrison, who was working on a project that I had at that time supporting automatic optimization...The nubbin of the debate was Steve's defense of not having to build optimizers anymore because the programmer would take care of it. That it was really a programmer's issue.... Seibel: Do you think C is a reasonable language if they had restricted its use to operating-system kernels? Allen: Oh, yeah. That would have been fine. And, in fact, you need to have something like that, something where experts can really fine-tune without big bottlenecks because those are key problems to solve. By 1960, we had a long list of amazing languages: Lisp, APL, Fortran, COBOL, Algol 60. These are higher-level than C. We have seriously regressed, since C developed. C has destroyed our ability to advance the state of the art in automatic optimization, automatic parallelization, automatic mapping of a high-level language to the machine. This is one of the reasons compilers are ... basically not taught much anymore in the colleges and universities."
-- Fran Allen interview, Excerpted from: Peter Seibel. Coders at Work: Reflections on the Craft of Programming
So it was time to embrace UB was means to recover ground, and here we are back at hardware memory tagging and sanitizers as ultimate mitigations for a patient that will never recover from its bad habits.
Who said memory corruption had to occur "reliably"??
"Permissible undefined behavior ranges from ignoring the situation completely with unpredictable results"
> compilers can't allocate local variables in register.
Why not?
> If it's on stack, out-of-bounds write may reliably corrupt stack, so how can it be on register?
"Permissible undefined behavior ranges from ignoring the situation completely with unpredictable results"
> And I am yet to see any good answer.
What would qualify an answer as "good" IYHO?
Usually when I write code that corrupts memory, it reliably corrupts memory an unreliable way.
That means I would gladly take a compiler that could not allocate local variables in registers.
Or even better, just do a bounds check. The standard does not preclude storing a raw pointer and a length together. Why couldn't the compiler do something like that and do a bounds check?
(Yes, the compiler would have to use its own version of libc to make that happen, but it could.)
In other words, I want correct code first, performant code second. I'll take the performance hit for correct code.
People claim that exploiting UB gives 1-2% speedup and that that's important. What they conveniently leave out is that you only get that speedup when your code has UB, which means that you only get that speedup when your code has a bug in it.
Fast wrong code is bad. Very bad. Correct code, even if slower, is an absolute must.
Assuming there is no UB and optimizing under that assumption gives a speedup. You get the speedup whether that assumption is wrong or not. If that assumption is wrong, your program might also blow up in hilarious ways. But in no way does making the assumption that your code has no undefined behavior rely on your code having undefined behavior.
Sure, but to make sure you do not have any UB, at least for things like signed arithmetic, you basically have to either give up signed arithmetic, which is what I've done by using only unsigned, or ensure your arithmetic is always within range, which requires formal methods. Otherwise, you have UB, and that 1-2% speedup is because of that UB. If you do what I do, you get no speedup. If you use formal methods to ensure ranges, you do, but at an enormous cost.
> But in no way does making the assumption that your code has no undefined behavior rely on your code having undefined behavior.
But it does. Because the optimizations that assume UB only kick in when UB could be present. And if UB could be present, it most likely is.
Once again, those optimizations will never kick in on my code that uses purely unsigned arithmetic. So yes, those optimizations do require UB.
That sounds rather extreme. I imagine it would be ruinous to performance. To my knowledge the various safe alternatives to C have no trouble with register allocation. Safe Rust and Ada SPARK, for instance, or even Java.
> The standard does not preclude storing a raw pointer and a length together. Why couldn't the compiler do something like that and do a bounds check?
That technique is called fat pointers. It's been well researched. Walter Bright, who just posted a comment elsewhere in this thread, has a blog post on it. [0] I imagine ABI incompatibility is one of the main reasons it hasn't caught on.
C++ compilers do something similar, storing array lengths in an address like myArray[-1]. They need to store array lengths at runtime because of the delete[] operator (unless they can optimise it away of course). C++ still allows all sorts of pointer arithmetic though, so it wouldn't be easy for a compiler to offer Java-like guarantees against out-of-bounds access. Doing so takes a sophisticated tool like Valgrind rather than just another flag to gcc.
Wouldn’t recommend relying on either of them for making your build correct. It breeds the false accusation that it’s the optimizations that break your code, when the code is in fact already broken and might fail for other reasons.
I'll take 5% or even 10% for correctness because, again, fast and wrong is very bad.
I thought the original spirit of undefined behavior was 'we have competing compiler implementations and don't have the political will to pick one'.
This doesn't seem right? Specific types of aliasing make optimization more difficult/impossible because they are specifically not UB. Rust's aliasing optimizations can (theoretically?) occur specifically because same-type aliasing is UB in Rust. I'm not sure how "UB prevents optimization" falls out of that?
... What do you call it then when you put aliasing pointers into pointer variables "defined not to alias"? Can you explain how this reduces UB?
But it's not the UB here that prevents optimization, it's precisely the fact that aliasing is (sometimes) defined. If it were always UB, the compiler wouldn't ever have to worry about it... but that'd require some interesting changes to the language to keep it usable. Definitely not backwards compatible changes.
This is equivalent to saying that pointer arithmetic is disallowed. Pointers are by their nature in the C virtual machine offsets into a linear memory space, so for any two pointers, x and y, there exists a c such that (ptr_t)x+c == (ptr_t)y, and thus there can always be aliasing.
Historically, many platforms - such as Multics or Windows 3.x - didn’t have a linear memory space, they had some kind of memory segmentation. The industry has largely moved away from that towards the flat address space model. Go back to the 1980s, it was still a much bigger thing, and people used C on those platforms, and the standard was written to support them. The actual detailed control of memory segmentation is inherently non-portable so cannot be addressed by the standard, but the standard defines pointer arithmetic in such a way to support those platforms - pointer arithmetic on unrelated pointers is undefined, because if the pointers belong to different memory segments the results can be meaningless and useless.
edit: allowing that would prevent even basic optimizations like register allocation.
edit: s/virtual/abstract/
>I honestly think trying to write a highly optimizing compiler based on a different interpretation of UB would be a worthwhile experiment. We sorely lack data on how big the performance gain of exploiting UB actually is. However, I strongly doubt that the result would even come close to the most widely used compilers today—and programmers that can accept such a big performance hit would probably not use C to begin with. Certainly, any proposal for requiring compilers to curtail their exploitation of UB must come with evidence that this would even be possible while keeping C a viable language for performance-sensitive code.
He doesn't know such interpretations are implemented by widely used C compilers.
> This example demonstrates that even ICC with -O1 already requires unrestricted UB.
The example demonstrates nothing of the sort. It demonstrates that ICC uses unrestricted undefined behaviour, not that this is required in any way shape or form. (The only way the word "requires" is reasonable here is that the behavior seen "requires" use of this kind of UB to be present. But that's something very different, and doesn't match with the rest of his use).
> writing C has become incredibly hard since undefined behavior is so difficult to avoid
No, it has become difficult because compilers exploit UB in insane ways. The platform specific UB that he claims is "not an option" is, incidentally, exactly how UB is defined in the standard:
Permissible undefined behavior ranges from ignoring the situation completely with unpredictable results, to behaving during translation or program execution in a documented manner characteristic of the environment (with or without the issuance of a diagnostic message), to terminating a translation or execution (with the issuance of a diagnostic message).
This was made non-binding in later versions of the standard, so optimiser engineers act as if these words don't exist.
And of course there is no reason given for this interpretation being "not an option". Except for "but I wanna". Yes, it's not an option if you really want to exploit UB in the insane ways that compilers these days want to exploit it.
But that's not necessary in any way shape or form.
> That would declare all but the most basic C compilers as non-compliant.
Yes, because by any sane interpretation of the standard they are non-compliant.
On a more general note, I find this idea that things are necessary because I want to do them really bizarre.
This is reminiscent of the argument in "How One Word Broke C" [0, HN discussion at 1]. I'm not particularly convinced this argument is correct. In my opinion, it's "ignoring the situation completely" that's the phrase of interest; after all, what is assuming that UB cannot occur but "ignoring the situation completely"?
I'm a nobody, though, so take that with an appropriate grain of salt.
[0]: https://news.quelsolaar.com/2020/03/16/how-one-word-broke-c/ (currently broken; archive at https://web.archive.org/web/20210307213745/https://news.quel...
For example:
int f(int* i) {
int val = *i;
if (i == NULL) { return 0; }
return val;
}
1. i is NULL. Program flow will be caught by the NULL check and the return value is 0.
2. i is not NULL. The NULL check cannot be hit, and the return value is val.
As allowed by the standard, I'll just ignore the situation with UB, leaving
> i is not NULL. The NULL check cannot be hit, and the return value is val.
Alternatively, I can assume that UB cannot occur, which eliminates option 1, leaving
> i is not NULL. The NULL check cannot be hit, and the return value is val.
You get the same result either way.
And that gets to the root of the problem: What exactly does "ignoring the situation completely" mean? In particular, what is the scope of a "situation"?
However, ignoring the situation completely in this case is emitting the code as written. This is not hard at all, despite all the mental gymnastics expended to pretend that it were hard.
That is the compiler's job: emit the code that the programmer wrote. Even if that code is stupid, as in this case. It is not the compiler's job to second guess the programmer and generate the code that it believes the programmer meant to write.
In this case:
1. Dereference i.
2. Check i.
3. If i is null return 0.
4. return the value from line 1.
Again, this isn't hard.
"Ignoring the situation completely" means ignoring the fact that this is UB and just carrying on.
If my program attempts to write outside the bounds of declared array, ignoring the situation (that this is UB) is letting that write happen, and letting the chips fall where they might.
How is assuming it cannot/must not happen, and then optimising it away because it did happen "ignoring the situation"??
It is not just "possible", but exceedingly simple to comply with what is written in the standard and generate code in that situation. For example:
Writing beyond the bounds of an array whose length is known is apparently UB.
int a[4];
a[2]=2;
a[6]=10;
What was impossible about that?
Seems to me it's not the opposite but the exact same thing.
int a[4];
a[2]=2;
a[6]=3;
"Assume the situation cannot occur": don't know, but according to the UB extremists the compiler can now do anything it wants to do, including not emitting any code for this fragment at all (which appears to happen) or formatting your hard disk (which doesn't, last I checked).
Assuming that a[6]=3 does not occur, despite the fact that it is written right there, would also allow putting a into registers, which "ignoring the situation" would not.
The workarounds and effort required to tell a compiler today that no, you really did want to do the thing you said might well be insupportable. I figure they started going astray about the time self modifying code became frowned upon.
Since almost 5 years by now.
For example, if signed integer overflow would be implementation defined behavior, then any weirdness would be limited to just the integer operation that overflows.
Lots of other stuff can be expressed as implementation defined behavior. That would probably kill some optimizations.
So the question is more, do we want a portable assembler? In that case as many C constructs as possible need have defined behavior. Either defined by the standard or as part of the compiler documentation.
Another possibily is to have standards for C on x86, amd64, arm, etc. Then we can strictly define signed integer overflow, etc. And say that on x86, pointers don't have alignment, so a pointer that points to storage of suitable size can be used to stored an object of different type, etc.
If the goal is to run SPEC as fast as possible, then making sure every program trigger undefined behavior is the way to go.
Another point of view on the same question: Looking at software and hardware, the latter evolved insanely, but the former didn’t get seemingly faster, at least in userlands. Why bother with UB-related optimizations at all for a wide spectrum of software? Is there even software which benefits from -O3 and doesn’t use vectorization intrinsics? Why can’t “we” just hardcode jpeg, etc for few platforms? Is that really easier to maintain opposed to maintaining never ending sources of UB?
Iow, why e.g. my serial port or ata or network driver has to be implemented in C, if data mostly ends up in stream.on(‘data’, callback) anyway?
How do you write "put 0x12345678 to register 0x04000001" in assembler? mov eax, 0x04000001 / mov [eax], 0x12345678
How do you write it in C-as-portable-assembler? You write (u32)0x04000001 = 0x12345678;
How do you write it in Java? You can't, the language has no such ability and if you try it's a syntax error. You have to call into a routine written in a lower-level language.
How do you write it in C-as-abstract-machine? You can't, the language has no such ability and if you try it's undefined behaviour. You have to call into a routine written in a lower-level language.
By the way, you can't write an operating system in C-as-portable-assembler either. No access to I/O port space, no way to define the headers for the bootloader, no way to execute instructions like LGDT and LIDT, no way to get the right function prologues and epilogues for interrupt handlers and system calls, no way to invoke system calls. All those things are usually written in assembler. Writing operating systems in C has always been a lie. Conversely, you can extend the compiler to add support for those things and then you can write an operating system in extended-C!
It was so easy and then blasted into something unbelievably complex in just few years. Maybe 80386 wasn’t a good place to run typescript-over-asm kernel, but do we still have this limitation today?
Arguably with their push for Swift and Java/Kotlin, Apple and Google are much forward than Microsoft on this matter, given that .NET tends to suffer from WinDev worshiping C++ and COM.
You can get that BASIC experience nowadays when playing with uLisp, MicroPython and similar environments for embedded platforms, most of them more powerful than 16-bit home computers.
For most languages, writing hardware trap handler becomes quite a bit of an issue. In trap handler you cannot rely on an extensive runtime system. Anything that does garbage collection is probably out. Anything that does dynamic memory allocation is out as well.
Beyond that, how easy is it to create pointers, create datastructures that match a specific memory layout, etc. Low level device drivers need to talk to hardware in very specific ways. If it is hard to talk to the hardware, most people are probably not going to bother using that language for an operating system.
In theory you could mix and match languages in a kernel. For example, a filesystem could be written in a language that has an extensive runtime system.
Of course, there are things that cannot be expressed in safe code in either language. But marking fragments of code as unsafe, where C-like unconstrained access is allowed, helps a lot to minimize such areas and make them explicit.
There is definitely room for a more expressive and safe languages in the kernel-level space. We can look at Google Fuchsia or maybe at Redox OS, both are very real operating systems trying to use safer languages, with some success.
Vs. the kernel must keep working reliably when resources are running low.
which sorts
Dynamic languages only do memory-related ops unpredictably, or is it more than that?
An app that runs has its own nicely mapped address space. And it interfaces with devices through system calls. An operating system has to keep the actual addresses of everything in mind, and it usually has to talk to devices through virtual addresses.
As an example of what I think might be the problem. If the OS wants to read data from a device, it might allocate a buffer, wait for the device to write into that buffer, and then later read it. For the compiler, that is essentially "reading uninitialized memory" and thus undefined behavior.
If a programmer could communicate this idea to the compiler, it would be somehow safer to write such code. There is a big difference between intentionally reading what looks like initialized memory, and doing so by an oversight.
It depends. If your hardware is behind a bus or controller device that's serviced by a separate driver, then you're using APIs of that bus/controller driver.
But think of having to talk to the TCP/IP stack using system calls - you are using an API but you'll still need to have some structure just beyond moving data back and forth over a bus. A USB mass storage driver is going to need different data moving over the USB interface than a USB network interface driver.
Different buses work differently as well - USB device addressing is different than SATA or PCI-E device addressing.
If you are really talking directly to a device, you're manipulating registers, bits, ports, etc. You may have to involve IRQs, etc. Your serial port, for example, can hold 16 bytes before it generates an IRQ to tell the CPU it has data if it's a 16550 I think. Your SATA interface doesn't work like that, it can actually DMA data directly to memory. But both of these could be streamable devices to an operating system.
Historically, it has been difficult to beat monolithic kernels for performance and efficiency, and through significant effort, monolithic kernel based OS's exist that are reliable enough to be useful. However, the monolithic kernel is not the only OS architecture.
Software has gotten much faster. Yes, almost all software benefits from -O3. What do you mean "hardcode"? as far as I know libjpeg can be linked statically...
UB is easy to maintain, lets take integer addition & overflow, you just issue an ADD instruction and however that CPU executes the ADD instruction is how integer overflow works on that platform and then in the C standard you write "integer overflow is undefined behavior".
Operating systems are written in other languages, such as C++ and Rust.
One requirement is that a language must be compiled and thus cannot rely on a runtime. That excludes Go and Java.
The language needs to support direct memory manipulation.
The compiled binary cannot be emitting system calls since that binary IS the kernel. Thus the compiler must be told to not link or include standard libraries.
You need to disable certain optimizations like advanced vectorization extensions and red zone use on the stack.
There are others. Lots of specific control needed.
Maybe I'm wrong, but I know that there exist CPUs made specifically to natively execute Java bytecode, so in reality if the hardware has a baked-in language interpretation it would be actually possible to write an OS completely in Java
Note that you can port any OS written in C to such a CPU with "just" a new compiler backend and a few drivers. Your OS won't take advantage of the features the CPU provides, but it will work.
Having said that, there have been OSs written in languages with heavy runtimes, even GC.
Only for those that cargo cult against using them.
https://www.f-secure.com/en/consulting/foundry/usb-armory
https://developer.arm.com/solutions/internet-of-things/langu...
Yes one can implement a CPU that natively executes a runtime for a high-level language, make your own ASIC, or FPGA, etc. that does this. That is a more advanced response to the general question.
Knowing the detailed points I mentioned will help understand why specialization of processors is needed to support other higher-level languages that do not meet the requirements I laid out.
To some small extent, yes. But I don't think that is the main issue here.
The real issue is that the stakes are much, much higher when implementing an operating system than when writing, say, an image editor. You can live with an occasional crash in a userland app. But the same crash in an operating system may open the door to taking over the entire computer, possibly even remotely.
How many operating systems do we use that are less than 20 years old?
A driver doesn't need to be implemented in C, but the kernel API is likely written in C, your code needs to talk to it somehow. If your driver is written in C, it's as simple as #include.
t = require(“awesome-8253”)
p = require(“pio-node”)
// cast your usual 0x42, 0x43, 0x61 spellsAlso may UB exploiting things like strict aliasing are enabled by default at all optimization levels in GCC.
I expect that most instances of implementation-defined behaviour come with additional rules which state that the implementation has to define something sensible.
https://stackoverflow.com/questions/18420753/unspecified-und...
Imo the way is to develop more tools that detect (either by analysis or at runtime) those bugs and run the code with those attached as often as you can afford it (to take the performance penalty).
There are quite a few idioms where overflow is used intentionally. There is no reason to turn that into undefined behavior if it works fine on the underlying platform.
For instance ARM, x86 scalar, and x86 SIMD all have different integer overflow rules for shift operations.
Platform specific behavior should be as rare as possible. It's a recipe for bugs. The rule is simple enough and major compilers have flags to prevent the overflow. Obviously you pay the performance penalty for using them. It's a choice you're free to make as it should be.
I know what UB is. I think the idea is brilliant and saves millions of dollars of burnt coal every day. Sometimes security matters more and then you compile your code with every flag/sanitizer you can find to exchange performance for security.
Because in reality as you observe so many overflows are unintended. Most of the time programmers, especially dealing with 16-bit or wider integer types, treat them as though they were mathematical integers, and so overflow is extraordinary and worth flagging.
Unfortunately UBSAN doesn't prevent you getting this wrong, perhaps somewhere important. If your test inputs never trip the corner case where overflow occurs you can switch off UBSAN in release builds, ship it, and overflow on a real system where it has real consequences.
This topic has turned up before. [0]
edit: I got the default behaviour the wrong way round here:
I think C# gets this right. Ordinarily it handles integer overflow by throwing an exception, but it has an unchecked keyword which gives you wrapping. [1] If you're writing code that is expected to wrap, you use the unchecked keyword, and you carry on using the usual arithmetic operators. (I believe you can also instruct the C# compiler to default to unchecked behaviour, so there's also a checked keyword. This strikes me as a mistake.)
Strictly speaking Java gives you the option of checked vs unchecked integer arithmetic, but with terrible ergonomics: the '+' operator always silently wraps, if you want throw-on-overflow behaviour you have to call a method. [2] This is of course so unsightly that Java programmers tend to stick with the infix arithmetic operators regardless of the wrapping behaviour.
C++ has templates and operator overloading, so you can use a library to get signed arithmetic to wrap, or throw, without undefined behaviour. [3] Such libraries are very rarely used, though.
See also this very good blog post by John Regehr, who specialises in this kind of thing. [4] To quote the post:
> Java-style wrapping integers should never be the default, this is arguably even worse than C and C++’s UB-on-overflow which at least permits an implementation to trap.
[0] https://news.ycombinator.com/item?id=26538606
[1] https://docs.microsoft.com/en-us/dotnet/csharp/language-refe...
[2] https://docs.oracle.com/en/java/javase/17/docs/api/java.base...
[3] https://www.boost.org/doc/libs/1_78_0/libs/safe_numerics/doc...
I would really like it for the C standard to clean up Undefined Behaviour. Back in the 1980s when ANSI C was first specified, a lot of the optimizations that modern compiler writers try to justify via Undefined Behaviour simply weren't part of many compiler's repertoires, so most systems developers didn't need to worry about UD and there was no push for the standard to do so as a result.
If people really want the optimizations afforded by things like assuming an int can't overflow to a negative number in a for loop, my personal position is that the code should be annotated such that the optimization is enabled. At the very least, the compiler should warn that it is making assumptions about potentially UB when applying such optimizations.
There is this false belief that all legacy code should be able to compiled with a new compiler with no changes and expect improved performance. Anyone who works on real world large systems knows that you can't migrate to newer compilers or updated OSes with zero effort (especially if there's any C++ involved). I understand that compiler writers want to improve their performance on SPEC, but the real world suffers from the distortions caused by viewing optimizations through the narrow scope of benchmarks like SPEC.
Perhaps people wanted int overflow to be UD because that is semantically clean. Code where possible integer overflow is intended and not bug is IMHO significant minority of all integer usage, in vast majority it is a bug, similar to invalid memory access. So it would be reasonable for some compilers/platforms to handle integer overflows by cpu exception translated to signal killing the process (similar to SIGSEGV for invalid memory access), and only integer operations explicitly marked for overflow (i.e. unsigned in C) be allowed.
It is just our experience that makes 'integer overflow should wrap' and 'invalid memory access should crash' default positions, while 'integer overflow should crash' and 'invalid memory access should be ignored/return 0' are alternative positions. Conceptually, both are just stricter or looser ways to handle code errors and it makes sense that C does not prescribe neither one.
That's a fair point and it irritates me too. But you're not forced to run -O2 in production if you don't want to. I've had projects where we compile with multiple compilers and different settings -- including different libcs -- just to get all the warnings, while we ship non-optimized or minimally optimized debug builds to the customer..
That's not an entirely absurd idea, especially if you wanted to enable optimization piece-wise for legacy code. But in the end does it matter whether you specify the optimizations you want in the build system or in the code?
> At the very least, the compiler should warn that it is making assumptions about potentially UB when applying such optimizations.
The assumptions are usually "assume no UB." I don't know what kind of warnings you expect but this idea has been brought up before and it's not good because it would just make the compiler flood warnings on any code that does any arithmetic on variables (as opposed to compile time constants).
Even saying Spectre causes you to care more about memory safety sounds like marketing fluff. Fixing memory bugs doesn’t even prevent Spectre issues.
But, officially, undefined behavior is always undefined, not just at higher optimization levels.
Additionally in C UB is often not intentional nor trivial to detect in your codebase since it may be the interaction of two pieces of code that are not anywhere obviously close to each other. There comes a point where faster but broken code isn't better it's just broken.
Compilers justify register allocation by assuming stack can't be smashed, contrary to hardware reality. Because smashing the stack is UB. That's what C UB is for.
Think of it like trading one cycle-accurate emulator for a better, more tightly-coded cycle-accurate emulator. Or a plain loop-and-switch bytecode interpreter for a threaded-code bytecode interpreter. The way your code is running on the underlying machine changes, but the semantics of your code relative to the abstract machine the code itself interacts with should not change.
> Compilers justify register allocation by assuming stack can't be smashed, contrary to hardware reality.
...which should be entirely fine, as the existence of a stack isn't part of the exposed semantics of the C abstract machine. Values can be "on the stack" — and you can get pointers to them — but nothing in the C abstract machine says that you should be able to smash the stack. There's nothing in the C standard itself describing a physically-laid-out stack in memory. (There are ABI calling conventions defined in the standard, but these are annotations for target-uarch codegen, not facts about the C abstract machine.) There could just as well be a uarch with only the ability to allocate things "on the stack" by putting them in a heap — in fact, this is basically how you'd have to do things if you wrote a C compiler to target the JVM as a uarch — and a C compiler written for such a uarch would still be perfectly compliant with the C standard.
If C were an interpreted language, the fact that the stack can be smashed would be referred to as a "flaw in the implementation of the runtime" — the runtime allowing the semantics of the underlying uarch to leak through to the abstract-machine abstraction — rather than "a flaw in the interpreter" per se; and you'd then expect a better runtime implementation to fix the problem, by e.g. making all pointers to stack values actually be hardware capabilities, or making each stack frame into its own memory segment, or something crazy like that.
As a compiled language — and especially one that has encapsulation-breaking holes in its abstract machine, like the C inline-assembly syntax — you can't exactly expect a runtime shim to fix up the underlying uarch to conform to the C abstract machine. But you could at least expect the C compiler to not let you do anything that invokes those cases where it's exposing potentially-divergent uarch semantics rather than a normalized C-abstract-machine semantics. As if, where there should be a shim "fixing up" a badly-behaved uarch, you'd instead hit a NotImplementedException in the compiler, resulting in a compilation abort.
I’m a little more familiar with the C++ committee than the C committee. The C++ committee prefers to declare something an error than to declare it undefined behavior. They only declare something undefined when they believe it would be unreasonably difficult for the compiler to detect the problem (e.g., using multiple, incompatible, definitions for an inline function; which can happen if you have a macro expanding differently in different parts of the code, or you have multiple definitions for the function, but only one definition is ever visible to the compiler at any moment in time).
I’m pretty sure the “signed overflow is undefined” rule is something of a special case: it should be easy to detect when source code doesn’t have a hard coded upper bound, but giving an error or warning in all cases will create too many false positives, and declaring that it wraps on overflow has been deemed unacceptable by the committee.
Sure, the "Why are you deleting my checks for if *this is null" is a little silly - but there are definitely sharp edges where UB conflicts with actually useful things you might want to do. Did you know seqlocks are undefined (benign race conditions)? Ever ran into padding concerns playing poorly with trying to do atomic CAS?
It's not unreasonable for the standard to say 'padding is undefined', 'data-races are undefined' - but having no way to say "hey, trust me, please un-poison this thing you don't like" is pretty unfortunate.
It wouldn't be undefined behavior if you used atomic variables--data races involving atomics aren't undefined behavior.
Unless things have changed a lot in the past two years neither LLVM or GCC do that much optimisation around atomics so this comes with disastrous performance implications as well as overhead battling the standard.
In practice (UB aside), this is basically fine in the context of a read, compute, CAS loop. Those bytes do have some value in the machine and if that memory isn't written they won't mysteriously change. It's playing games with the optimisers and UB however. You might be able to get around this by first initialising the bytes to zero, then in-place copy constructing whatever you want? I wouldn't bet anything serious on that being defined though.
(I don't know if that's the case, but it's what I thought the GP was implying.)
We have reached the point where C programmers cannot understand the language/compiler anymore.
Given that this has been going on for a long time, my hope is that Rust will be the next systems programming language.
https://github.com/rust-lang/unsafe-code-guidelines/blob/mas...
The only winning move is not to play. (with C)
This loose mapping between four optimization levels and a variety of obscure language situations is invisible and situated only in a handful of top engineers' memories. That's not a great way to run a language if you actually care about the correctness of programs written in that language.
I used to think that higher levels are for giving the compiler more time to think so it can come up with a better result (and for somewhat specifying the desired space-performance tradeoff). In my mind the "don't do it if you don't know what you are doing"-things were seperate flags, like -ffast-math.
Obviously I eventually learned that bad things can happen, but "just build prod with -O2" still seems to be what almost everyone does.
If there were, it’d be better for the compiler to tell the programmer about them rather than rediscover them every compilation.
That optimisations must not change the visible behavior (except for running time) of a program was the cardinal rule of optimisation.
Then it got flushed down the toilet. (With the trick of retroactively redefining the behavior to be undefined in the first place and thus all bets being off).
- O1 includes optimizations that only require a small amount of additional compilation time. (Fast compilation, code is much faster than -O0.)
- O2 includes O1, and additional optimizations that use a medium amount of compilation time. (Medium compilation, reasonably fast code.)
- O3 includes O2, and additional optimizations that take more compilation time. (Slower compilation, faster, larger code. Unrolled loops and the like.) It often makes sense to compile hot inner loops with O3 because the compiler can do some fairly aggressive vectorization, if you know what you are doing... and the code remains correct in either case, if it was correct in the first place. It's basically "free" vectorization, when it works.
- Os includes O2, except flags that increase code size. (Medium compilation, code is slightly slower and smaller than O2.)
The only optimization level that actually changes the meaning of correct code is -Ofast, which is rarely used. It's described in the manual with the phrase "Disregard strict standards compliance".
ALL levels of optimization will change the meaning of your code if your code relies on undefined behavior. For example,
https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html
> -faggressive-loop-optimizations
> This option tells the loop optimizer to use language constraints to derive bounds for the number of iterations of a loop. This assumes that loop code does not invoke undefined behavior by for example causing signed integer overflows or out-of-bound array accesses. The bounds for the number of iterations of a loop are used to guide loop unrolling and peeling and loop exit test optimizations. This option is enabled by default.
It's always enabled.
Citation needed? AIUI optimization levels are entirely up to the compiler. GCC's man page says nothing about the safety of these levels.
> Optimizations aren't supposed to change the meaning of your code.
And they generally won't. The trouble is when the meaning of your code wasn't well defined to begin with (or you thought it meant something it didn't).
Well, it used to.
"Do what I mean!" "Well, if your code does UB then it doesn't mean anything"
In fact, the behaviour of code is only limited to defined behaviours... if you stray... then it's undefined.
Sorry, no can do. If that's the contract than C is a completely useless language.
That is the contract. Your inference may be correct, but it does no good to insist that C should behave differently than it does. There are better options.
The license is still the closest thing you've got to a contract with the makers of Linux.
Same with a language specification.
That would require kernel devs to radically rethink how they interact with I/O, which would probably require specific architectures.
In other words, writing a kernel portable on any of the existing ISAs that is also performant is basically impossible, barring some humongous breakthrough in compiler technology.
Seems to me that when it comes to brass tacks, UB is kind of the "we are all adults here" engineering tradeoff that enables shipping fast and useful software, but is technically not strictly defined and thus usually does what you want, but can result in bugs.
I'm convinced that the original UB rule was intended to capture this, and the wording was an error later seized by compiler developers. As evidence, consider Dennis Ritchie's rejection of `noalias` as “a license for the compiler to undertake aggressive optimizations that are completely legal by the committee's rules, but make hash of apparently safe programs”². If anyone at the time had realized that this is what the definition of UB implied, it would have been called out and rejected as well.
this could be violated even by simple optimizing compilers that do constant propagation or strength reductions.
Again, if you do not want optimizations -O0 is always available.
"To help ensure that no code explosion occurs for what appears to be a very simple operation"
How does constant propagation cause code explosion?
It kinda boils down to what exactly you mean by defined behavior. A C programmer's take might be that you can run a conforming program in an emulated abstract machine and get defined results out of it. And then you can run the same thing on real hardware and expect to get the same result (modulo implementation defined behavior). This definition leaves some things out (e.g. performance, observable effects in the "real world") but it captures the computational semantics.
Another programmer's take might be more akin to a portable assembler. In that case, you certainly could define reads and writes for arbitrary pointers, in the sense that they must cause corresponding (attempted) loads and stores at the machine level. However, the definition wouldn't be complete since it inevitably leaves much to the underlying implementation. Thus you could have "defined" C programs that show completely different behaviors depending on which implementation and hardware you used. It would be impossible to say what the program's output must be "in the abstract." For someone who just wants to output assembly, maybe that's fine. I'm not sure other people would be too satisfied with it. An out of bounds write could still blow up your program and be remotely exploitable; practically the same thing as undefined behavior, except that now your compiler is also barred from optimizing.
There's quite a bit of tension between these two camps.
Alternatively, you could fully define it at a great runtime cost and potential exclusion of real hardware implementations.
It is probably impossible to make a low-level language with no implementation-defined behavior, but it is certainly possible to make one with no undefined behavior. For example, you can put in your spec that overflowing an unsigned integer can give any value; that is different that putting in your spec that it doesn't happen and if you write it the variable might have no value, multiple values, or burn your socks off.
It would be great if this wasn't possible, because then you wouldn't have a whole class of security vulnerabilities.
On older systems without memory protection, the situation is even worse - you could scribble all over the OS memory as well and who knows what happens then.
But apart from that, you can have pointers, with many levels of indirection, as long as there are rules that prevent use-after-free, unsynchronized concurrent access, and other UB-worthy problems. Rust's borrow checker with rules for no mutable aliasing and Send/Sync markers for concurrent access comes close, but it has to give up on generality for safety (e.g. it can't reason about circular data structures).
With untyped pointers, yes. But it seems to me that if you have strong typing for function pointers you could mostly avoid that.
> UB is kind of the "we are all adults here" engineering tradeoff that enables shipping fast and useful software, but is technically not strictly defined
Well, no, of course it isn't -- the clue is probably in the first half of the name, "Undefined Behaviour"... ;-)
> If the content source operand is 0, the content of the destination operand is undefined.
It makes much more understandable why linux codebase is riddled with compiler extensions, ISO C is simply not reliable anymore.
The issue is bigger than what trembles on the surface, just like Dennis Ritchie said, it is a timebomb, soon enough these nuances will burst into a big issue in linux kernel, or worse yet, some essential system like avionics.
There is a necessity for some form of UB in a C-like language, and that has to deal with pointer provenance. You see, in C, everything lives in memory, but on real hardware, you want as much to live in a register as possible. So a compiler needs to be able to have reasonable guarantees that, say, any value whose address is never taken can never be accessed with a pointer, and so can be promoted to a register. As a corollary, this requires that things like out-of-bound memory accesses, or worse, converting integers to pointers (implicating pointer provenance here) need to have UB in at least some cases, since these could in principle "accidentally" compute an address which is the same as a memory location whose address was never taken.
That suggests that the problem isn't UB per se. If we look at the two canonical examples, arithmetic overflow and strict aliasing, we can see that one of the features of these things is that they have a pretty obvious well-defined semantics [2] that can be given for them, and furthermore, there's no way to access these well-defined semantics even avoiding this feature altogether. And I think it's the lack of this ability to work around UB that is the real issue with C, not UB itself.
[1] For example, it is UB to pass in rand as the comparison function to qsort. I'm sure many people will not realize that before I wrote this, and even parsing the C specification to find out that this is UB is not trivial. For an interesting challenge, try giving a definition of what the behavior should be were it not UB--and no, you can't just say it's impl-defined, since that still requires you to document what the behavior is.
[2] I will point out that, for arithmetic overflow, this semantics is usually wrong. There are very few times where you want <large positive number> + <large positive number> = <negative number>, and so you're mostly just swapping out an unpredictably wrong program for a predictably wrong program, which isn't really any better. However, the most common time you do want the wrapping semantics is when you want to check if the overflow happened, and this is where C's lack of any overflow-checked arithmetic option is really, really painful.
Some comments: - we have proposal for provenance rules (N2676), which would clarify the rules and make sure that pointer-integer roundtrips work (now broken on some compilers due to an "interesting" interpretation of provenance) Pointer-to-integer round trips could then be used to work around provenance-based optimizations if you need to (and fix existing code with those casts).
- for signed overflow, you can get run-time traps with the right compiler flags and this is IMHO far better than defined wrap-around for unsigned.
- C23 will get checked integer functions based on the GCC builtins
- you can get (relatively cheap) run-time bounds checking by using VLAs (some people think VLAs are generally bad, but pointers to VLAs are a bounded pointer type!)
There is of course much left to be done. But I like to stress that UB gives the compilers room to do what it wants. But this does not have to be optimization. If programmers do not like what they get, they need to complain to their vendors! The standard has to follow existing practice, so this has to be driven by the compiler vendors based on the feedback of their customers. So please, if you want to see changes, tell this to your compiler vendor and do not accept "we do this because the standard allows this" as an answer.
For [2] I disagree for two reasons. For one, merely having an answer at all, rather than entering an undefined behavior is valuable; having x+y=pseudorand(x,y) would still be better than complete UB because the damage that can be done is limited. The other reason is that addition/subtraction/multiplication does actually work modulo 2^n, so if I write e.g. (a+b-c) where b can be large but b-c is known to be small, it works even if the language is technically left-associative and having the a+b first is UB.
I pretty much agree with your point that the main problem is there is no way to work around UB; if there were a workaround there would be much less reason to complain. I would go a bit further and say such a workaround needs to be in the code itself to be effective, not merely a command line arg like fwrapv, so that it gets kept around when doing header inlining, LTO, copy-pasting code to other projects, etc.
I am guessing, for example, that you are not aware that every compiler will optimize `printf("Hello, world!\n");` to `puts("Hello, world!\n");`. memcpy and memset are even more heavily-optimized function calls by the compiler: for example, they don't prevent memory-to-register promotion.
There is a canonical workaround - cast operands to unsigned integer, do the operation, cast it back to signed integer. Overflow during casting to signed is implementation-defined behavior instead of UB.
Being _predictably_ wrong is actually much better than being _unpredictably_ wrong, in that case, it's not UB(unpredictably) anymore is it?
If the semantics are exactly what you saying you could just convert the predictable wrong value to a correct one and you have overflow checking.
Pretty much every non-legacy architecture does IEEE floating point. Pretty much all of them do a flat address space. The word size is some power of 2(32 bit, 64 bit, maybe 128 bit in the future). They are almost always little endian. The memory models are converging towards the C++ memory model.
Given that, I think simplifying the language and getting rid of foot guns could be done without losing any significant performance or actual flexibility/portability.
This is what the OpenBSD team did to OpenSSL. If the code has some complexity that is only necessary because it might have been run on a VAX or AIX or early Cray architecture then it is time to excise that complexity. They deleted thousands and thousands of lines of support for architectures that are only seen in museums and landfills today.
However architectures of the future aren't going to have flat pointers (PAC/CHERI etc), and even current architectures have some non-IEEE floats (https://en.wikipedia.org/wiki/Bfloat16_floating-point_format / https://en.wikipedia.org/wiki/Unum_(number_format)).
https://www.microsoft.com/en-us/research/wp-content/uploads/...
Is there any reason for modern C to still support anything else?
Guaranteed order of evaluation of arguments almost made it into the standard, but because of regressions, we didn't quite get the full benefits; for example: i = i++ + 2;
is now fully defined, while this:
f(++i, ++i);
Ideally for every UB taken away we would get one or more pragmas to get the optimization back like ivdep. The issue is that that doesn't help old code bases.
> The issue is that that doesn't help old code bases.
D has a flag called -revert=issue where old behaviors will be supported for a time.
It would be useful to have another type variant that offers defined overflow (like in unsigned) together with signed range for such cases. But it still makes sense for basic integers to have overflow as UD, as in most cases it is not expected behavior.
Note that in current C, if one needs defined overflow on signed integers, one can cast them to unsigned, to the operation and cast result back to int. That makes it implementation-defined instead of undefined.
Yeah, but the gotcha happens when it is expected behavior.
C23 will require 2s-complement.
Signed overflow is still UB in C. I think this is a better choice than wrap around, because overflow is often a bug. With UB, you can use static analysis (to some degree) or run-time traps (if the compiler supports this) and then fix those bugs. If it were defined to wrap around, those bugs are much harder to find.
But that’s basically always been the case. I doubt you could stay within the first ISO C standard and write a modern operating system.
For example, 4.4BSD, early Solaris have just about everything you would expect in a modern operating system. And give the age of those systems, they were written in early versions of the C standard.
The C standard has notes about a freestanding implementation, mainly for developing kernels. The requirements C has for freestanding environments are very limited ( http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf , you get float.h, iso646.h, limits.h, stdalign.h, stdarg.h, stdbool.h, stddef.h, stdint.h, and stdnoreturn.h) and the starting point for the program is implementation defined; you might get _Atomic, but that’s optional). For the record, C++ goes a little overboard and promises all sorts of things in its freestanding environment.
https://groups.google.com/g/comp.std.c/c/zzyii-DlMiU/m/RnH5i...
The issue was, do modern operating systems require newer versions of the C standard? Or can you write everything in C89.
Same applies to using C in hardware that lacked memory mapped IO.
Do you take that view because of some intrinsic deficiency in the language, or the difficulty of memory safety, or something else?
https://groups.google.com/g/comp.std.c/c/zzyii-DlMiU/m/RnH5i...
No. The standard specifies what's undefined, optimization levels don't change it (though there are compiler flags such as -fwrapv which make undefined things defined).
However, turning off optimizations will make behavior easier to predict.
Try asserting that they're not NULL in glibc and try to boot your machine! Oops... bad compiler people, bad!
Just one example, I can't be bothered to post others.
Besides that, how would you specify the behavior of reading from an address the user randomly generated himself? What is stored at an address not generated by the compiler is out of scope for the compiler. If you try to define something like 'always trap' or 'always return 0', you incur massive overhead.
Besides that. The fact that signed integer over/underflow is undefined behavior actually unlocks a lot of reasonable optimizations that treat integers like actual integers. Things like (a+1 > a) always being true.
It is very hard to specify the behaviour of, for example, use-after-free, or accessing an automatic variable after it goes out of scope, or data races, in a C-like language without significant runtime cost.
She does embedded systems at Anduril using Rust.
No, most operating systems that people are actually use are written in ISO C, so the headline is by definition wrong.
"For example, a well-known security issue in the Linux kernel was produced by a compiler incorrectly assuming a pointer null check was unnecessary ([40] fig. 6) and deleting it as an optimization. Or consider this (simplified) patch report for Linux [25]: The test [for termination] in this loop: [...] was getting completely compiled out by my gcc, 7.0.0 20160520. The result was that the loop was going beyond the end of the [...] array and giving me a page fault [...] I strongly suspect it’s because __start_fw and __end_fw are both declared as (separate) arrays, and so gcc concludes that __start_fw can never point to __end_fw. By changing these variables from arrays to pointers, gcc can no longer assume that these are separate arrays."
Both of those sound like simple bugs in the compiler optimizer implementation, in which case they could hardly be use as examples of how the current C standard was bad.