If you for a second put yourself into the shoes of a person who thinks "the apocalyptic stuff" has even a 5% chance of literally happening in the real world, you might see how you wouldn't agree to move on from it.
The much more likely explanation to me is that people are just spitballing, either because they've watched too much sci-fi, or they have some weird counterintuitive agenda (e.g. Anthropic and OpenAI trying to position themselves as the amazing, trustworthy keepers of this dangerous technology before their IPOs).
All there really is is arguments from theory (most notably, the orthogonality thesis and instrumental convergence). It's not like we have a spare universe to conduct experimental research on the likelihood of human extinction. That said, the 5-10% range is not some specific person's wild guess, it's the median wild guess of a poll of AI experts: https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/
> or they have some weird counterintuitive agenda (e.g. Anthropic and OpenAI trying to position themselves as the amazing, trustworthy keepers of this dangerous technology before their IPOs).
They are not "trying to position themselves" that way - back when those companies were founded, the people who did it were already convinced they were trying to make the ("inevitable") development of AI go better. It's not some novel galaxy-brained marketing strategy, it's just what they thought the whole time.
But the discourse is dominated by paper clip experiment discussions and not let's say by the fact that new grads have an unprecedented difficult time getting jobs. Unsurprisingly one of those is a sexy hypothetical beneficial to power and the other one is not.
Multiple problems can be important, pointing a different one out doesn't invalid or take away from another one.