There's plenty we could do about it if we knew an asteroid was coming. With years of warning, we could use nuclear explosives to deflect it. With days of warning, we could at least evacuate affected areas.
In fact, we are plenty good at imagining disasters that could conceivably happen. Most of them never do; if we put our efforts into imagined disasters, we wouldn't have time to do anything about the real ones. The way we tell which is which is, necessarily, to wait and see what actually happens, then take steps to guard against similar disasters in the future. Reality is smarter than we are. We need to pay more, not less, attention to what it tells us.