It's more than harmful—prior to the latest common usage, toxic usually meant potentially lethal for sufficient dosage.

So, while it is indeed free of connotations of intent, it evokes disgust and, in fact, invites ostracism. If something is so dangerous that it can kill you, there's rarely much wisdom in attempting to talk it out of being lethal. Better to just remove it from your environment.

In any case, I agree that the word is tragically overworked. And don't we all know that overwork can be toxic.