Look, as this article demonstrates, it's not hard to build a backdooring compiler. Even if you want to build it in a more robust way than checking filenames, it's really not difficult: it's (admittedly complex) pattern matching, and quite a lot of optimization in fact boils down to pattern matching. The problem is that the pattern matching you'd need to do to get the everything-is-backdoored scary effect is brittle as fuck.
Compiler output tends to be effectively nondeterministic. I mean, the goal of the compiler is to produce completely deterministic input, but very subtle changes can have cascading consequences. (I say this as I am trying to fix a test for LLVM's opaque pointer changes). Even something so simple as figuring out how to make bit-equivalent reproducible builds with the reproducible builds initiative took a few years to really get going, since there are so many things that are effectively random that you wouldn't consider at first (e.g., iterate over all files in a directory).