His association to Klaus Fuchs[2] was probably more suspect than Oppenheimer (despite how suspicious they were of him) given he was allegedly an actual spy (I say allegedly because I know how suspicious of the FBI people are around here, and not without reason in some cases).
According to the Feynman Bio I'm reading right now [3], Fuchs was said to have joked (when Feynman wasn't around) that he would be the most likely person to be a spy because of how often he left the base alone to visit his sick wife (or so he said).
[1] http://www.atomicheritage.org/history/soviet-atomic-program-... [2] https://en.wikipedia.org/wiki/Klaus_Fuchs [3] https://www.amazon.com/Genius-Life-Science-Richard-Feynman/d...