Nazi's are an easy target. The problem starts when things are not as clean cut -- when actual politics are involved, as opposed to crude stuff like "exterminate all jews" and "blacks are inferior", etc that Nazi's might say.
Let's say a team wants to overthrow the presidency and bring something like another parliamentary system in the US. Or rewrite the constitution and remove the BS ideas from 3 centuries ago.
Should they be allowed to express their ideas? If they get somewhat popular and other people worry about their influence?
And what's stopping disagreeing people from considering them the same, or worse, as they'd have more current influence, than Nazis, and asking to ban their speeches etc?
(Many people considered the CPUSA like that in the 30s and up to the 60s)