I would add that this statement might equally apply to government - that is, if you disagree with the policies of a local government (property tax too high, school district underperforming, etc), move somewhere else. If you disagree with your state, move to another one. And if you disagree with national policies, immigrate to a country whose policies you agree with. But therein lies the pitfall of freedom of association - if you disagree with your HOA, or your local/state/federal government, you may not have the financial or legal means to change your situation.

I see a dichotomy of "associations with oversight" and the converse (associations with no oversight). If your HOA forces you to build a fence, there is no appeal process similar to if a government entity charges you with a crime (e.g. appeal to a state/federal circuit court and ultimately to a supreme court). Within each is an enforcement mechanism that may or may not work. In your dinner party example, which is an association with no oversight, a naked table dancer may simply crash the subsequent gatherings. In all levels of government, there are those who continue to hold their position regardless of the misdeeds and accusations levied against them.

Perhaps that is the essential trade-off of freedom, to be free to associate with any group of people despite the possibility of being victimized by them (now or in the future).