I think this is an important point. I found the Intercept's article[1] about the ACLU defending Charlottesville protesters' free speech right pretty interesting in that regard: Some people insult and blame the ACLU for defending free speech rights of even the most infamous people, but as the article points out, people fail to understand that if a precedent is set on them, it's diminishing everyone's rights.

The ACLU understands this, which is why they defend such people.

[1] https://theintercept.com/2017/08/13/the-misguided-attacks-on...