This isn't some free-speech issue where you debate politely and sip iced water and other frippery.. this is actual people killing other people. This is how the the real world actually operates, instead of libertarian-nerd theory world.
And you know Nazis would be extremely violent people because no rational person would self identify with that group, so already they're batshit insane, which means they're likely to be extremely violent. And sure enough, when hundreds of Nazis gathered this weekend in Charlottesville, you actually ended up with an event measured in terms of "death toll".
We have to treat these people like armed and dangerous criminals, like you would ISIS or any active shooter.
And we all need to understand that government limits speech in many, many ways, not just the "fire in the theatre" example, but with things like sedition and other criminal conspiracies to more mundane things like copyrights and libel.
People forget that we went to war against these people and used to kill Nazis wholesale less than 80 years ago, because the Nazi party went to war against America. Identifying with them means you've actually declared war against the US. Not sure how much clearer you could be in declaring yourself to be a violent and dangerous criminal than that.
Just ban them. Arrest their members. Don't be the socially inept libertarian nerd that thinks only in terms of theory without any real-world experience. It's perfectly fine to limit rights and freedoms in the real world. You can do it!