A great many people disagree and do not think that is a likely or even remote outcome. If they are correct, then you are shouting "FIRE" in a crowd for no reason.
If there's a lot of smoke coming, people are running out of the building and you can see an ominous red glow in the windows, shouting "FIRE" is the right thing to do even if we are not going to be engulfed in flames this very second or the next. The potential costs given the evidence we all have are simply not comparable.
What smoke? So far the predictions of AI x-risk folk haven't panned out the way they said it would. In fact the opposite has happened. What smoke are you referring to?