Free speech is granted to ensure that oppression can be beaten by language, not to permit people to use language to systemically oppress.
Free speech, as defined by numerous Supreme Court decisions, protects speech that expresses ideas, however distasteful, but does not tolerate advocating of action that will cause harm, nor statements whose sole purpose is to invoke an altercation--"fighting words".
The clear distinction from Oliver Wendell Holmes is that you can yell, "Fire!", but not in a crowded theatre. Free speech is not without limits.
So you can legally say that someone is ugly, but you cannot advocate assaults on people that you perceive as ugly.