This doesn’t work either. The consequence of extinction is infinity (to humans). Likelihood * infinity = infinity. So by hand-waving at a catastrophic sci-fi scenario they can demand we heed their demands, whatever that is.
In some cases like asteroids, you can look at the frequency of events, and if you manage to push a big one of of your path then you can say the system worked.
But is much more difficult to measure a system that didn't rise up and murder everyone. Kind of like measuring a bio-lab with a virus that could kill everyone. You can measure every day it didn't escape and say that's a win, but tells you nothing about tomorrow and what could change with confinement.
Intelligence represents one of those problems. AI isn't going to rise up tomorrow and kill us, but every day after that the outlook gets a little fuzzier. We are going to keep expanding intelligence infrastructure. That infrastructure is going to get faster. Also our algorithms are going to get better and faster. One of the 'bad' scenarios I could envision is that over the next decade our hardware keeps getting more capable, but our software does not. Then suddenly we develop a software breakthrough that makes the AI 100-1000x more efficient. Like lighting a fire in dry grass, there is the potential risk for an intelligence explosion. When you develop the capability, you are now playing firefighter forever to ensure you control the environment.
[Robert Miles AI Safety]
Lets take "big asteroid impact" as baseline because that is a credible risk and somewhat feasible to quantify: Probability is somewhere under 1 in a million over a human lifetime, and we barely care (=> we do care enough to pay for probe missions investigating possible mitigations!).
So the following requirements:
1) Humanity creates one or more AI agents with strictly superhuman cognitive abilities within the century
2) AI acquires power/means to effect human extinction
3) AI decides against coexistence with humans
Only need 1% probability each to exceed that probability bound. And especially 1) and 3) seem significantly more likely than 1% to me, so the conclusion would be that we should worry about AI extinction risks...
If you accept that fact that extinction has finite negative utility, it's completely valid to trade off existential risk reduction against other priorities using normal expected value calculations. For example, it might be a good idea to pay $1B a year to reduce existential risk by 0.1% over the next century, but might arguably be a bad idea to destroy society as we know it to prevent extinction in 1000 years.