I say, it’s “rarely a compiler bug.” When teaching new developers I remind them that you should assume the bug is in the code you just wrote before jumping to the bug being a compiler, framework or OS issue. It’s just a matter of probabilities.
At the end of the day, you want to optimise for debug time. There's the probability that it's a compiler/developer bug (respectively very low/very high) and the time it takes to rule it out. It's of course best NOT to start by investigating the compiler bug.
1) With with an esoteric environment that has relatively few users.
2) Have specific objectives that require a lot of edge case testing or ridiculously thorough fuzzing of the binary.
3) Go out looking for them by crafting nifty things that aren't much used.
Otherwise it would imply large companies (Google-scale) experience thousands of compiler bugs per year...
This seems likely to be true, from my experience.
Fortunately, most compiler bugs I run into (mostly with gcc and llvm) are not code generation bugs (which can eat months of debugging), but just segfaults / rejecting correct code / other broken stuff.
These features are much more likely to have buggy edge cases in the compiler. And only a small select group of programmers will run into those bugs over and over again.
Eg complex template metaprogramming - which had known broken corner cases in all C++ compilers up until a decade or so ago.
Large companies with compiler teams want to use them, so they do update their compiler, so they will find bugs in it.
Btw, what do you think this prints with clang? (Whether the answer is a compiler bug is debatable…)
printf("%#x\n", 1 << 32);(Spoiler: it prints uninitialized stack memory. A bit closer to demons flying out your nose than you'd expect!)
On Darwin/x86_64, this actually means that it prints out bits from an "uninitialized" register, specifically %rsi in this case, since that's where the second argument should be (even for variadic functions).
Certainly a jarring failure mode! However, if updating your compiler causes you to run into this, you already have a bug in your code (one that clang -- arguably not forcefully enough -- already warns you of).
I find them because of a combination of : - a very large codebase - compiling with optimizations (O3) and targeting recent archs - yearly compiler upgrades - an extremely extensive testsuite
I have found all kinds of bugs (frontend, middle, backend) - the codegen ones tend to be very nasty to diagnose.
Not that most problems still aren't in my code, but I've increasingly run into framework/library level issues and very rarely compiler issues. Can't say I've triggered any kernel issues yet.
Networks can and do go down of course. But in my experience, the vast majority of these issues are actually the result of something extremely mundane like a typo in a hostname. I've resolved an incredible number of issues over the years by simply reading the error message someone sent to me and asking them to check the exact thing that the message says is wrong.
Compiler bug or CPU issue, take your pick :)