> none were Soviet citizens who came on work visas
Many of them were from countries that were at war with the United States, like Germany and Italy.
George Gamow, one of the most important American physicists ever, was born in Russia, educated in the USSR and began his career there, before defecting to the US. He didn't work on the Manhattan Project, but he did consult on other military projects during WWII. There are lots of other examples of Soviet-American physicists, including one of the inventors of quark theory (George Zweig) and the one of the people who formulated perhaps the most famous paradox in quantum mechanics (Boris Podolsky).
> Surely this is something to think about before creating a caste of foreign high skilled workers and academics?
Casting suspicion on students and scientists in the United States because they come from "enemy" countries is not a healthy thing for society to do. I also object to your use of the word "caste." Castes are hereditary groups with defined roles in society. What we're talking about here are simply smart people who did well in school, and managed to get into top universities.