So let me throw out an idea that might help justify this trust too. Compile the same TrueCrypt sources with a totally different compiler, then use both binaries in a deterministic way and compare the raw encrypted result. (I'm assuming here that the same encryption keys and data will give the same result, but don't know for sure if that's true.)
(= ((a-binary b-source) b-source) b-binary)
If you trust a-binary, and you trust b-source, then if that returns true you should be able to trust anything created with b-binary. (a-binary b-source) will not equal b-binary, but ((a-binary b-source) b-source) should.If the hardware is not executing binaries as described, then all bets are off. Compiler A could, in theory, be build and run on a homebrew CPU (http://www.homebrewcpu.com/), but if you are putting that much effort into this, this better be your hobby...
Even if that were not the case, if the hypothesis is that GCC was compromised at some point in the past by a shadowy organization, then you have to consider the possibility that this shadowy organization also got to the other compilers. I think that is where probability steps in though; how confident are you that at least some of the compilers are still safe (or perhaps, at least compromised in conflicting ways)?
If you are able to inspect the actual contents of the program, not the output of a program that itself inspects the actual contents of the program, then this problem disappears. You have to examine the machine code without an intermediary program that could lie to you.
(Of course it is very unlikely that IDA Pro, objdump, or even 'od' is compromised in this way, but I would say this class of attack is largely hypothetical and implausible already...)
Edit:
From wikipedia: "What's worse, in Thompson's proof of concept implementation, the subverted compiler also subverted the analysis program (the disassembler), so that anyone who examined the binaries in the usual way would not actually see the real code that was running, but something else instead."
Which the author does, in fact:
"Of course, we need to trust the compiler, but in this case, it is independent of TrueCrypt."
He even links to the "trusting trust" article.
Because TrueCrypt is several orders of magnitude more high-profile of a target (being actual cryptography instead of a compiler) and probably also several orders of magnitude easier to compromise in a useful fashion & spread.
Paranoia is without bound. Who knows, you yourself might be a sleeper agent and you just don't know it yet! Maybe your eyes really were compromised in development! So you approach the problem from the point of view of what is likely, and what is not. You cannot guard against every single paranoia, but you can guard against ones you deem more likely.
The compiler backdoor would have to be very old to exist.
Lets all just stop right here. No, dont mention that cross-compile with different compilers and compare. Just everyone stop.