I think that intelligent people have always observed that human beings are their own worst enemy most of the time. Outside of extreme cases like the great plague of Europe, human history has mostly been a history of humans being the major threat to other humans. People noted how warfare was democratized with the advent of the pile head crossbow, and it's true they panicked. The same thing kept happening, warfare kept becoming broader, the means to kill increasingly democratized, and every step along the way people panicked and thought that this was the end.
Then we developed nuclear weapons, and everyone finally realized the end was in sight, no need to panic, the threat was finally existential and real. Our response to this realization was to mutually stockpile enough nuclear weapons to destroy most human life and all human civilization many times over. Still, like the crossbow nuclear weapons have not totally destabilized society, and now intellectual people are looking for the next threat, and for the first time all of them are potentially world ending. That doesn't mean the world will end, as we've seen many times in the past, but as you say the gap is narrowing, and we're never going back to a time and we don't have the means to actually end it all.
I think people worry about robots because it's the most comfortable thing to worry about, compared to nukes and climate change at least.