There has to be a give and take, otherwise this logic can justify many things we've decided is wrong. Things for example like firing someone trying to unionize or for being black or a woman. There is a difference between freedom of association for an individual and freedom of association between a company/organization and its employees or members and that is part of why.