Given how aggressively compilers (including Clang) are exploiting undefined behavior these days, it is surprising to see Clang developers (I presume) on the bug being so blasé about introducing undefined behavior themselves.
I don't see anyone being blasé about this. The issue comes down to something really simple:
- The compiler can emit substantially simpler code if it has a guarantee from the platform's implementation of memcpy.
- The compiler authors thought they had such a guarantee and so chose to leverage it.
Now, maybe they don't have that guarantee on all platforms. If that's the case, its flat out and simply a bug in the compiler. And you'll find folks on that bug arguing that it doesn't actually hold, which is good and important.
But it's actually quite reasonable to have a guarantee from a platform's libc that this will work and leverage that. The compiler isn't generating C code or code bound by any standard. This is about two parts of the implementation of C agreeing about what internal invariants will hold.
The problem this bug highlights (and it is a real problem, and not one anyone takes lightly) is that we have insufficient agreement about these invariants, both for certain platforms (glibc) and even within LLVM. Fixing this remains absolutely critical, but may only involve documenting the agreement.
I should have clarified that the comments I was referring to were on this bug: https://llvm.org/bugs/show_bug.cgi?id=11763
I was surprised to see: "I don't think it's really worth warning about, considering I don't know of any real-world implementation where it doesn't work."
That sounds a lot like someone knowingly writing UB in a C program because no compiler they know of breaks the intended behavior. I doubt any memcpy() has ever guaranteed in any kind of official way that overlapping regions are ok, even if it has historically worked in practice.
I mean I'm sure it'll get fixed now, I was just surprised to see this kind of reasoning coming from someone who I presumes works on the compiler.
btw. I'm pretty sure I heard that glibc broke this behavior sometime recently. There's a new memcpy() symbol in glibc these days with a different symbol version that no longer provides correct behavior in the case of overlapping memory regions. This is all second-hand from some other people I was working with, but if I got their meaning correctly I think this issue will actually have relatively widespread impact.
We (the Clang community members, and I suspect the GCC community members as well) have also worked to change language standards to provide guarantees relied on consistently and where the lack of guarantee has no reasonable basis. Also that comment is very old, and predates us learning about all of why this is probably something actually worth warning on. That discussion ended up in a different forum, and we do actually warn on stuff like self assignment.
Understand that it is often precisely the people working on a compiler that do deal with the cases of when we discover widespread UB in real code and whether or not the code is incorrect and must be changed or the standard is incorrect and must be fixed.
But all of that is about a warning, and not about the actual question of how the compiler and the system library interact.
Note that both Clang and GCC have at least some variants of this bug, and so I don't actually expect glibc to break this in the foreseeable future. But we're still actively working to defend against this on platforms for which we don't have a guarantee.
That seems a separate issue though from what calls the compiler should emit. I am not following why LLVM would emit memcpy() when the semantics it actually requires from libc are memmove(). Am I missing something?
What is more frustrating about modern compilers (and this is not limited to LLVM) is the desire to squeeze out a tiny bit more performance by exploiting undefined behavior. I'm thinking of things like eliminating calls to memset that "don't seem to do anything user-visible," eliminating code based on signed overflow being undefined, and so on.
Most of the the undefined behavior that exists in C is there because different machines used to do different things. For example, there used to be non-two's-complement machines. It wasn't put there because C was somehow designed to be hyper-optimized. People who relied on the undefined behavior could legitimately make the same argument that you are now making-- that it worked fine on their machine, which they knew very well.
I am curious why nobody has brought up the idea of making memmove as efficiently handled as memset, rather than using undefined behavior here. UB feels like the wrong way to go.
Would you have said this if Ulrich Drepper was still in charge of glibc? Under his ownership glibc broke all sorts of things that happened to rely on memcpy behaving like memmove[1]. I'm not sure he would have been any more accommodating of non-conforming compilers than of non-conforming flash implementations.
You are perhaps thinking of this: https://sourceware.org/bugzilla/show_bug.cgi?id=12518
I'm not sure if that change broke glibc memcpy for identical arguments or just ones that were different but overlapped.
This quote suggests that Drepper wouldn't have had any problem breaking things that passed identical arguments to memcpy:
>The existence of code written by people who should never have been allowed to touch a keyboard cannot be allowed to prevent a correct implementation.
- Programmers can and do emit substantially simpler code if they have a guarantee from the platform's implementation of <feature x>.
- Programmers think they have such a guarantee. (I don't know a single programmer around me who knows the complete list of undefined behavior, myself included)
- Compiler injects bug -- and LLVM/CLang seek excuse in the "technically not a bug" completely useless answer.
- Sane projects actually disable all that kind of crazyness (adding -fno-<everything> options every now and then) so all that kind of cute optims are useless in the real world (either dangerous, or completely disabled)
It's actually reasonable to have a platform that e.g. gives you 2's complement arithmetic or defines the result of shifts, or allows unaligned access, because mainstream computer hardware does provide those guarantee. And at least half of C programmers are not even aware those kind of stuff are technically forbidden, (historically often because of other architectures, some obscure, and not compiler optimizations, btw). More knowledgeable people who know about the spirit of the standard (1) can also be bitten by compiler revelation of bugs because compilers devs are doing literal and insane interpretation. Bug ensues, including security holes.
(1): Let me remind you about the definition of undefined behavior: "behavior, upon use of a nonportable or erroneous program construct or of erroneous data, for which this International Standard imposes no requirements" One keyword is nonportable and it is clearly not considered enough. So yes, there are numerous constructs that are reasonable to expect on a given architecture when we do not learn the compiler manual by heart, but some people don't give a fuck about the principle of least astonishment when they can gain 1% on a synthetic benchmark.
ARM, PPC, and x86 all define the results of some shifts differently (for shift amounts >32UL) and behave differently with unaligned pointers (only x86 tends to accept them). Which mainstream hardware are you using?
Adding magical nice semantics to your language is another way to hide bugs in your program. Try building with ubsan and doing runtime testing instead.
2) Compilers leveraging undefined behavior knowledge get a lot of hate for breaking this implicit contract defining what they "should" do as expected by programmers. Unfortunately these feelings are rather misguided, compilers are just doing what they can to produce the best code for you using the explicit contract that exists: the language specification(s). Are these specifications perfect? No, but they're what we have. If someone were to define a new language that actually captures the assumptions "real" programmers "expect" when they write code then compilers could choose to support it (and probably would!) and everyone would be happier. Until then this is a big case of "how come you're not doing that thing you never said you'd do and no one is willing to define properly?" which is a bit silly. Imagine you're a compiler developer: what assumptions CAN you make to satisfy everyone? In short, compiler developers aren't your enemy, they're just doing everything they can to produce the best code they can for you.
2a) Use of -f* is in a way used to define that a special variant of a language is desired, for example -fwrapv builds the code using an unnamed variant of C where signed integer overflow is well-defined.
[1] http://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html
I have suddenly lost a great deal of respect and confidence for LLVM/Clang after reading this bug thread; I value software reliability above all else, and being able to write reliable software without the compiler negating that effort. It seems I can no longer be assured this is even a priority of my compiler any more, with Clang. All the more reason to switch to Rust I suppose.
That the Clang people are trying to deflect this bug as "fine, no fix necessary" is kind of disgusting software engineering behavior. Compilers spontaneously introducing undefined behavior in another layer (to the point of causing codegen warnings from perfectly correct C code, is as clearly cut a BUG as any, independent of current API implementations outside of the ISO spec.
If you want to make undefined API calls the norm, then make a new API where the behavior you want is defined. Invoking undefined behavior of a decoupled library that users can swap out is flat out indefensible.
That they are reluctant to fix this makes me shy away from ever using Clang/LLVM, which is sad because before I had always held them with very high respect.
The folks working on other platforms seem universally unhappy with that bug.
And neither Clang nor LLVM are generating platform agnostic code. The code only has meaning for the specific platform being targeted.
Yet they do exactly the same thing, and pretty much casually. And I actually don't really blame them for doing it. Everybody makes mistakes. But I blame them for the double standard, the risks, and the attitude. The problem in the first place: engineering is also about mitigating the fact that mistakes will happen, often mitigating even the mistake of others, and the modern approach in regard with undefined behavior does not help and is even ultra-hurtful for that. I don't see how stability will emerge with that kind of thought process.
Myself and other compiler authors I work with are much more specific in their requests. We do actually define some behavior that the standard doesn't require, and we do document this and tell our users its fine to rely on it.
Unfortunately, we also have a lot of really clear experience with this stuff that lead us to two conclusions:
1) When we identify UB in code and go look at that code, for very large bodies of code what we find are overwhelmingly bugs. Bad, complete bugs. Defining the behavior wouldn't fix the bug, it was just one of the many symptoms.
2) We have real hardware differences for a lot of simple things that make them hard or impossible to give defined behavior for. As an example, shifting left by more bits than the size of the value gives different results on several different platforms.
None of this means that we don't need to help programmers cope with this. We do indeed also make mistakes, and in fact it was these compiler authors that built a tool to help programmers find UB in their code: http://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html
Hard or impossible to give _performant_ defined behavior for.
> As an example, shifting left by more bits than the size of the value gives different results on several different platforms.
You could, for instance, always mod the shift by the word size, but possibly at a significant cost.
Rust is backed by llvm.
> I have suddenly lost a great deal of respect and confidence for _LLVM_/Clang after reading this bug thread;
However, do you believe that there is little overlap between llvm developers and clang developers?