I had to leave an instance because the maintainer said pro-life views were forbidden. Didn't matter whether you expressed them or not - the owner had a policy against private opinions.
I agree with your point about freedom of association, but let's not pretend this is just limited to racism.
And if you support the concept of free association, what do you do when it collides with civil rights law?