Freedom of association (or more specifically: the freedom to not associate) has limits; for example, freedom of association "doesn't matter" in the context of whether or not it's legal or moral to prohibit someone of a protected class from patronizing a business.
you bring up an interesting point, freedom of association is also important, and it seems like those two are potentially in conflict over speech. I'd love to continue this debate, but HN's increased throttling of replies-to-replies makes it complicated. Can we continue over email? (mine is in my profile)