The problem with AI is that we might get an artificial general superintelligence before we get the first big disaster, and when a big disaster happens at that point, it could be so big that it’s catastrophic for humanity.

Lots has been written about this topic, https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-annou... is one of the more pessimistic takes. I wrote a more neutral introduction a while back at https://ruudvanasseldonk.com/2024/ai-alignment-starter-pack.