I disagree.
Freedom of speech, as in the legal right protected by the First Amendment in the United States, is indeed only against the government.
But free discourse, as an ideal and virtue for the citizens of a republic, is a much broader concept. The citizens of a democratic republic should be open to hearing new ideas and engaging in reasoned debate with each other.