I need to attract you, continuously, and then, in the same fashion, _hurt you_. Both elements need to be present for something to be toxic. If someone is "just" attracted, that's simply recruitment, and not toxic in itself.
This is precisely the case with cultish groups like neo-Nazis.
They draw vulnerable/disaffected people in, they have attractive elements (sense of community, us-versus-the-world, positive feedback on traits others see as negative) that make it hard to leave, and they hurt you (job loss, friends and family disown you, reinforcement of those negative traits).
The key thing is, sooner or later just like a pond full of toxic mining waste, the toxic waste seeps through and eventually the dams fully break.
We've seen exactly this in politics - it began with Trump and others spreading "birther" conspiracy myths in the Obama time, progressed with the Tea Party and culminated in a full-blown coup attempt where mass casualties were only barely averted.
If you just let a pond of toxic waste accumulate and don't do anything to clean it up, you will inevitably head towards disaster.