The hundreds, maybe thousands, of people working under him. The majority of which would need to agree that project "Dr. Evil style giant space rock randsom" was a good idea.
Still, I have to disagree with your point. Lots of people did bad things on Hitler's orders. Lots more looked the other way. The "thousands of people" argument falls apart when looking at historic examples.