Exactly. Taken to the limit if you extrapolate how many future human lives could be extinguished by a runaway AI you get extremely unsettling answers. Like the expected value of a .01% change of extinction from AI might be Trillions of quality Human lives. (This could in fact be on the very very conservative side, e.g. Nick Bostrom has speculated that there could be 10^35 human lives to be lived in the far future which is itself a conservative estimate). With these numbers even setting AI risk to be absurdly low, say 1/10^20, we might still expect to lose 10 billion lives. (I'd argue even the most optimistic person in the world couldn't assign a probability that low) So the stakes are extraordinarily high.

https://globalprioritiesinstitute.org/wp-content/uploads/Tob...