You're right that there are limits (just as there are limits on free speech), and I don't think any sensible person is advocating for freedom of association on the basis of religion/race/gender. The original comment was proposing that merely holding an ideological position should afford some sort of legal protection - this is akin to firing someone for political affiliations, which is allowable in most cases for private enterprise, with greater latitude for political activity afforded to government employees.