When Oppenheimer witnessed the first explosion of a nuclear weapon, he quelled his ethical reservations over the destructive power of his creation with a verse from the Bhagavad Gita [1] often mistranslated as the deity stating "I am death, destroyer of worlds", but more accurately - "I am time, and I will destroy these people with or without your involvement".
Had the scientists of the Manhattan project (Oppenheimer, Fermi, Szilard, etc) subscribed to the EA philosophy, they would have been unlikely to work on nuclear weapons development, and millions more would have likely perished in a land invasion of Japan. However, millions of Southeast Asians and South Americans did perish in the subsequent "proxy wars" of the Cold War era, so you can make a convincing historical "what if" either way.
Effective altruism is not a very useful philosophy if you don't actually know what is best for humanity. Oppenheimer's philosophy (the Gita philosophy) was to simply do his job without being attached to the outcome.