> We're killed by the things we spend a lot of time with, not random out-of-context bacteria that's never seen a mammal before.
We're mostly killed by things that have never been in humans before because we haven't developed defenses. Sure, most of those things don't kill us, but you really only need the right one to come along and basically wipe us out. Part of the reasons we're ultimately able to develop a defense is because it is Earth life and we've seen all the tricks. Any alien microbe that is able to establish itself inside our bodies might not have to worry about any of our defenses.
I think you've really got the wrong model. Think of the introduction of non-native species to various ecosystems. A lot of the time, the non-native species either can't hack it, or they find an equilibrium. But some of the time you get kudzu or eurasian watermilfoil.
In any case, the point is largely moot. Any civilization we are likely to meet in the next thousands of years will necessarily have to visit us and if they can do that, they can just lob a couple space rocks at Earth and wipe us out from a distance.