This is precisely the case with cultish groups like neo-Nazis.
They draw vulnerable/disaffected people in, they have attractive elements (sense of community, us-versus-the-world, positive feedback on traits others see as negative) that make it hard to leave, and they hurt you (job loss, friends and family disown you, reinforcement of those negative traits).