He ended up not being the decisive factor, but as the above said, he advocated for a nuclear first strike on the Soviet Union, and was already part of the nuclear decision-making process - he was invited to participate in the decision for which cities to strike in Japan but was ultimately overruled because he suggested to hit target with no military importance and high concentrations of civilians. He thought (very wrongly) that coexistence with the Soviets was impossible and that they had to be eliminated as soon as possible.

Here are sources if you want:

"Von Neumann, four other scientists, and various military personnel were included in the target selection committee that was responsible for choosing the Japanese cities of Hiroshima and Nagasaki as the first targets of the atomic bomb. [...] The cultural capital Kyoto, which had been spared the bombing inflicted upon militarily significant cities, was von Neumann's first choice"

"He was quoted in 1950 remarking, "During a Senate committee hearing he described his political ideology as "violently anti-communist, and much more militaristic than the norm". If you say why not bomb [the Soviets] tomorrow, I say, why not today? If you say today at five o'clock, I say why not one o'clock?"

(https://en.wikipedia.org/wiki/John_von_Neumann#Nuclear_weapo...)

I don't think that I need to give sources as to why nuking the USSR would have had a huge negative impact.