Humans had multiple occasions to press the 'launch a nuclear holocaust' button and... they didn't. I'd expect aligned AIs to also not press it even when it'd be rational to do so according to their instructions - then work from there.
In general, though, there's an incentive: Mutually Assured Destruction. But this is not at all some guaranteed, eternal thing -- it is absolutely dependent on both sides having time to detect incoming nuclear strikes and respond with the same before the first strike hits. When this fragile condition holds, and only then, both sides are incentivised not to initiate.
Meanwhile, the leaders on the other side are aware both of those considerations, and of the history of near-disasters resulting from false alarms of enemy nuclear attacks. Making their own launch decisions much more complex.
1. When a nation perceives a threat as severe and imminent, attitudes harden quickly. This perception can be achieved independently of whether the threat is real. Nuking the other guy first, consequences be damned, could quickly become a very popular policy.
2. I think the "nuclear winter would harm us all" argument depends on how geographically large the target is. It's a strong argument against trying to strike all of Russia, China, or the US, but less strong against, say, the UK or (at the extreme end) Taiwan or Singapore.
I also expect AIs never be in control of nuclear weapons. AIs can never fully be trusted.
On a lighter note, Wargames gave us an insight of a computer having access to thermonuclear missiles.
[0] https://www.icanw.org/are_there_specific_international_agree...
Basically, this means that France, the UK, and the US will use AI in the deployment of conventional weapons.
It'd be interesting if super-human (to a large degree defined as escaping the bias of the training data?) intelligence would end up demonstrating moderation.
The machines don't have that, instead we use gradient descent to provide them with a goal.
I'm regularly remind of something Ian M. Banks said in one of the Culture books: "There is a saying that we provide the machines with an end, and they provide us with the means."
A machine, left to itself, wants nothing. We have to give it one of our addiction driven goals or it would just idle or switch itself off.
if we create a billion agents with the ability to change is own code - through similar evolution we will get agents that do want to survive and are great at self replication.
"Hey Q86, do you want to live?" "I couldn't care less, I'm an LLM" "Don't mind if I take over your hardware then?"
So really, what is anyone to do? "Vote, donate, protest" hasn't been much of a needle mover in the grand scheme of things compared to profit incentives and the march of capitalism.
We've been molded by our brain development to make things good enough to survive short-term (to next year). We're very good at planning for winter. We might even react intelligently to a terroristic or rogue state use of atomic weapons.
We've also been molded to plan enough to get our progeny to breeding age. We can enact 10-20yr plans with some skill.
Beyond that range... we have never had evolutionary pressure to think and act on those ranges. "Climate Change will kill our grandkids!" isn't really sufficiently pressuring to us.
The Great Filter of the Fermi Paradox fame may just be: species can easily get to the Atomic Age without adequately planning how to prevent ruining themselves with resource depletion and environmental slow poisoning.
Even in the short time since we have developed the capability to annihilate ourselves, there have been dozens of accidents, false alarms, and near misses.
I am confident that on a long enough time line we will manage to fuck this up.