So they have a strong bias towards imagining the most catastrophic scenario.
In any case, what's the probability at which a possible extinction event becomes a risk worth taking? Even if there's just a 1% likelihood of current research bringing about a superintelligent AGI, and just a 1% likelihood of that AGI causing an existential catastrophe, no rational person should accept the risk, unless it was clear that not accepting it would yield an even worse outcome.
This is like me saying "Even if there's just a 1% likelihood of me getting struck by lightning..."
You're implying that 1% likelihood is the floor, because 1 is the lowest natural number and feels like a good default "low percentage", but there's zero justification for putting the floor that high.
Conjuring an unsupported "low" probability and multiplying it by a massive outcome to make it seem significant is one of the most irritating ways people launder their opinions(/gut feelings) through "math".
With regard to the second probability, P(AGI | current LLM research), it would certainly be overconfidence bordering on arrogance to assign it a value significantly lower than 0.01.
It's fine to eyeball it and give your gut feeling, but it's disingenuous to present that as probabilistic fact.
Edit: Also, you've downgraded your claim from "Extinction event" to "dangerously unaligned"
I imagine saying you're developing skynet is better than saying you are developing a very cool, extremely expensive to run chatbot that can do math and code.