Social consequences inflicted by other free people are those things which free people (i.e., those who are not incarcerated) enact upon you within their rights. These most exist in the negative: This is the right to not shop at your establishment; the right to not hire you to work for them; the right for them to not be near you, engage with you, or otherwise pay attention to you.
A simple way to think about it is that people whose opinions are far from the mainstream have to put in the intellectual work to convince people to associate with them if they value that opinions expression more than their own cohesion inside society. This is fair, as it's been demonstrated effective for various groups.
By misconstruing free speech to mean that the government ought to force private institutions to give everyone a microphone skews this work. It amounts to the government forcing private individuals to sponsor things they disagree with, and removes the ability of free people to disassociate themselves from that speech.