Long story short: historical accident. Modern safe systems languages like Rust and Ada¹ avoid undefined behavior unless you explicitly ask for unsafe code, and even then there aren't many _undefined_ things you can do (violate strict aliasing I guess).
Most undefined behavior could be implementation-defined, but the edge cases from porting C+Unix from the PDP-11 (which I understand to have been... quirky) to many other systems happened before the spec, then the spec authors had to make the Ideal C Machine a sort of subset of all the existing behaviors. Thus, some things that could've been implementation-defined ended up totally undefined (god knows what computers did on signed overflow back then—the compiler may not have been able to guarantee anything: hence undefined) and we've kinda stuck that way ever since.
That said, the people who shame the compiler writers for "exploiting undefined behavior to optimize microbenchmarks and look good" annoy me to no end. That's just how the spec turned out, put up or shut up (or use a nicer language that isn't trying to stab you in the back as soon as you let down your vigilance...).
Here's a fun history of C: http://pastebin.com/UAQaWuWG
¹ Strictly speaking, I recall the Ada spec has some wording similar to undefined behavior, but it's not something you hit often.
> even then there aren't many _undefined_ things you can do
For anyone curious about seriously using unsafe code in Rust (which is to say, using unsafe code in Rust at all, in any capacity), I recommend reading through the Rustonomicon: http://doc.rust-lang.org/nightly/nomicon/
You need to import a virtual package and some times also make use of specific pragmas.
Ada, and Modula-3 are very much "into your face" in what concerns unsafe code.
For example, if you intend to do a conversion between data types without guarantee that the target can can hold a valid data representation, you need to import Ada.Unchecked_Conversion.
And do something like this:
http://www.adaic.org/resources/add_content/docs/95style/html...
Same applies to everything else deemed unsafe.
Edit: To add to my reply - for example, Java has almost no undefined behaviour. (The single exception AFAIK is some kind of multithreaded data race situation). The cost paid is that e.g. bounds checks must be done (in general) on array accesses, as the behaviour is defined to be an exception thrown in that case. Similarly for null reference dereferences.
I am coding since the mid-80's, always been a fan of Algol lineage of programming languages, with C++ being the only exception to it, as it allows many similar features.
Never have I had a situation where better performance thanks to undefined behaviour was an advantage for the use case we were trying to solve.
I think outside games, HPC and winning compiler benchmarks there hardly an advantage exploiting it.
That's not true. Load-load forwarding benefits tons of code that isn't "games, HPC, or compiler benchmarks".
If the solution delivered is within the time constrains required by the customer, any time spent optimizing ms out of the application is just wasted money.
For example most CPUs wrap from most positive to most negative on integer overflow, but one kind might just clamp the value at the most positive value. Or give an error, so forcing them to behave in a given way would make it hard to write efficient C programs on some platforms.
Recently some compiler writers decided that they could interpret undefined behavior to mean "Do anything we like".
So for example a sane compiler on x86-64 would compile this :-
int test(int x)
{
return(x > x + 1)
}
Personally I think this was never the intent of the standard and compiler writers are abusing the meaning of undefined in order to sneak in optimizations but realy they are just producing compilers that are less and less trustworthy as they no longer do that people want or need.
C was always a language that was intended to be used across multiple hardware targets and different processors have different behaviours. Restricting behaviour on certain arithmetic operations might require slow code to 'work around' the processor and generate compliant code. The creators didn't want that so specified as little as possible, leaving the rest to be 'Undefined'.