You're right that people are more likely to use words like 'toxic' when they disagree. But your generalization goes too far, and the cases where it fails are important. I moderate posts on HN all the time whose views I personally agree with. That's probably the most important capacity a moderator needs. Some of those posts I would definitely call toxic.

To pick an example that won't upset any people (ok, not many people), if someone shows up in Common Lisp threads on HN with the sort of chauvinism that destroyed comp.lang.lisp, that's toxic and I'm going to moderate even if I share their views about Lisp. This is a real example btw. I found https://news.ycombinator.com/item?id=9813527, but there have been a bunch of cases.