What Linus correctly points out is that when multiple sources of entropy are combined, no single source can act to diminish the sum total of entropy. Even if rdrand was comprehensively backdoored and entirely predictable by the US government, it's still going to improve the quality of randomness. At worst it's a wash, but it will never reduce it.
Here's the direct link to Linus Torvalds' entertaining observation:
http://www.change.org/en-GB/petitions/linus-torvalds-remove-...