Freedom of association is rooted in freedom of speech. Otherwise all the Civil Rights Acts would be unconstitutional as they fundamentally impact association, but not in a way that dilutes speech which is what freedom of association covers, for better or for worse. So, social media corps lacking freedom of speech rights in their users content, they also lack freedom of association rights with regards to that content and those users. To push the metaphor too far, their users' speech and their own speech are like oil and water: they are too different to dilute each other.
Now, would an alternate universe in which freedom of association was actually a robust right on its own be a better one overall? Would I happily trade that for this social media thing? Quite possibly, quite possibly.