No matter how evil some group is (may they be pedophiles, satan worshippers, nazis, whatever...) silencing them and assaulting them is a crime and is against freedom of expession. The problem with making these exceptions like "Let's have freedom of speech but not for these and these people...yadayadayada" is we can never be sure where to draw the line.
(Neo-)Nazis are sure dumb as hell but as long as they have peaceful protest and they don't harm anybody physically (unlike their counter-protesters) it doesn't matter. And don't even start with the car-thing. An individual psycho does not compare to organized widely accepted violence.
If we were to ban nazis and far-right organizations because they are racist and apparently a "threat" then what about anarchists? They also are extremely violent and want to overthrow the government. (and in the US officially categorized as terrorist threat) What about BLM which is openly racist and violent (July 2016 anti-white/police sniper attack, riots, assaults, highway blockings etc.) If we start going down this slippery slope will have shitloads of organizations and ideas to ban.