Free speech as a concept isn't just about the first amendment and isn't just about the law. Free speech can be defended on any number of grounds, and many of them apply equally to privately enforced speech restrictions. For example, consider John Stuart Mill, one of the founding figures of modern liberal thought. In On Liberty, his spirited defense of free speech, he argued that shunning or widespread ostracism for expression of particular views was a danger, and would ultimately harm a society that engaged in it, even if the views being suppressed were obviously wrong.
"Free speech is only first amendment" is a shallow and ignorant view of the intellectual tradition of free expression.