Allowing the use of the courts to prohibit slander is a government action. Slander and libel are long standing exceptions to free speech. "Fighting words", espionage, contempt of court, or insubordination in the military are other exceptions that come to mind.
The "the first amendment only applies to the government" is an argument that anyone is free to choose not to associate with you, or consider you a jerk, for something you said.
"Congress shall make no law..." We've extrapolated a bunch from there, but the founding words are pretty clear.
Example: I specifically prohibit peanuts in my kitchen, because I serve meals to allergic children. You sneak peanuts into the kitchen. I sue you and win.
Does this mean "Peanuts are illegal"? No. I have a civil action and we have a system of civil courts wherein these grievances can be aired and adjudicated.