They
were allowed to speak. They
did speak. "Freedom of speech" applies only and strictly to governments, not to a private institution. And that's because the only way "freedom of speech" matters is if others are free to react to you as they please; Harvard might not like you and might tell you to get the hell out of their (and my, as it happens) neighborhood.
If you don't like that, then you have options. One, don't be cretinous. Two, hide it better. Three, accept that being cretinous will cost you opportunities and own it.
Actions have consequences. You don't get to hide behind "my free speech". Own what you do.