Most of the US based scientists were either US citizens (some natural born; others naturalized) or came to the US in the 20s or early 30s. Even then very, very few of these men were German citizens. Fuchs, who was one of the few German refugees, was part of a contingent from the UK. The only reason why Fuchs was even allowed to be a part of the project was because of political pressure outside the project.

This is all documented in exrutiating detail in Rhodes' series on the making if both the atomic and hydrogen bombs.

Was it? The Germans and Austrians involved, for the most part, left for the US years before the war, often by receiving academic positions in neutral or allied countries. Leaving during the war would naturally be treated with greater suspicion.

But those weren't doing house calls and were under guard. :)