For better or worse, that's how societies work. They're composed of individuals who do things based on their own values.
Suppose you're an employer. Would you want to hire a neo-Nazi who has visible swastika and Hitler tattoos? Would you put this guy in front of your customers? Probably not; your business wouldn't do too well. So when you decline to hire such a person, you're exercising your freedom of association, but also helping to condemn the neo-Nazi to homelessness and poverty. In some countries, he might be able to get some social assistance so he doesn't turn to crime, but it'll probably still be poverty-level.
Societies aren't just a bunch of people all doing and saying whatever the heck they want. There's consequences to your actions and your speech. If people like you more, you get better jobs and do better socially. If people don't like you, then you become an outcast. This can be good or bad: if the overall attitude is something awful, such as the idea that some people should be enslaved, then you get a society where lots of people are horribly oppressed. If the overall attitude however is that oppression is bad, then people who promote oppression (like neo-Nazis) are punished by being ostracized, and ideas like that are made unpopular and kept from spreading too much.