Generally speaking, we’ve held as a society that the rights of free association are very strong, but not absolute. We only allow the government to abridge these rights when the state has a compelling interest in doing so.
Obviously you’re aware that we have decided as a society that there are certain “protected characteristics” that cannot be the deciding factor for free association, including race, religion, sex, among other things. These are selected due to their immutable nature (religion is a special case I’m not going to explain here), and we’ve decided that it’s worse for society that people be excluded from economic or social participation based on who they are than it is for the government to override the free association rights of those who wish to discriminate.
Political affiliation and speech is not a protected characteristic, because it is a choice. You can choose to not only change parties, but also change how you communicate such things. In fact, many institutions fundamentally demand the ability to exclude people based on their political affiliation, including primaries and political groups. We literally could not run a primary, caucus, or PAC without the ability to discriminate based on political affiliation.
To continue the metaphor, this is less like someone getting kicked out of a shop because of their race or gender, and more like getting kicked out for refusing to wear pants.