Freedom of speech means freedom of speech you don't agree with. There is no need to protect speech most people disagree with. FYI, the ACLU has also protected the same people you refer to. The problem with taking away civil liberties, is that eventually it will come down to losing your own.
For example, if you want the government to shut down speech it doesn't like, what happens when opposing views are in power, and Donald Trump decides he doesn't like what YOU are saying. Do you really want to go there?