This is an incorrect analysis of where the conclusion comes from.

Allowing the use of the courts to prohibit slander is a government action. Slander and libel are long standing exceptions to free speech. "Fighting words", espionage, contempt of court, or insubordination in the military are other exceptions that come to mind.

The "the first amendment only applies to the government" is an argument that anyone is free to choose not to associate with you, or consider you a jerk, for something you said.