Sure — but life is a risk tradeoff, ie cost versus benefit. So we should focus our attention on the most dangerous things, not panic over everything.
There’s also contrary examples to your general rule, eg, catching cowpox protected you from smallpox — so you were better off being exposed to the relatively weaker cowpox, even if it wasn’t risk free.