I often wonder why I am so confident in my prediction of AI doom despite other intelligent people not being as convinced, and honestly I think it's just that I've been working with AI and building neural networks at a low level for well over decade.
I'm far from an expert when it comes to the frontier, but I feel like I have had to developed a strong intuition for these systems because it used to be fucking hard to get AI to do anything useful, but what you find is that the bottlenecks are literally always the same – compute, data and time
Algorithms matter, but mostly just because of efficiency.
For example, a convolutional neural network can very efficiently be trained to solve computer vision problems, but there's nothing stopping you using a much worse network architecture then throwing a ton more compute, data and time at the problem and getting the same result.
Algorithmic improvements are really just what we've needed to get around current compute, data and time bottlenecks. AIs like Deep Blue required a lot of algorithmic thinking back in the day to beat Kasparov in chess, but now we're so much less compute or data constrained that you could beat Kasparov without needing to put even a fraction of the algorithmic effort in if you just throw enough compute and data at it.
The reality is, AI doesn't fail. They may have appeared stupid to users for a long time but that wasn't because they didn't work, but because we were compute, data or time bottlenecked.
As soon as you assume compute and data will continue to grow, then it's really just a matter of time before all problems that are solvable with enough intelligence are solvable. Add significant algorithmic improvements every ~5 years to that equation and suddenly god-like AI seems like it's probably reasonably close.
The only question to ask then is what happens after you've created all these super-human AIs. And here I'm much less certain, but I see so many low probability risks in an ASI world that I'd bet everything on one of those bad things happening relatively soon after such AIs exist.