Even if rdrand is backdoored it would have to be a significant supplier of entropy in the resulting random number for this to be a meaningful attack vector, as soon as you mix it with other (good enough, large enough) sources of entropy you get a situation where some other attack is more likely to be far more feasible than to use the knowledge about some of the bits that rdrand contributes to the entropy pool.
Such as:
- good old b&e and placing a keylogger or hardware bug
(very easy to hide in a keyboard)
- a compromised bit of the OS
- compromising the application that you use to encrypt your messages or finding a significant weakness in the application.
- doing any of the above with the recipient