I think that the "association" in "Freedom of association" means the right to get with people (associate as in form a group).

The "association" that you speak of is a different concept: figurative association (how people compare or liken groups one to the other).

I do not think that it is right to conclude something about the one based on the other.