The difference isn't the violence at any one rally. The difference is that one group (the one with the swastikas) actually advocates genocide. As in: there were people in Charlottesville who openly told reporters that they want to send jews/blacks/muslims to gas chambers. And even if there were only a few of those, the swastika, "heil X", and nazi salute are undeniably linked with the history of the Nazi party, the holocaust, and WW2. I just scrolled through a few pages of pictures, and I think it's fair to say that the protest were pretty homogenous in that regard. I don't see many history professors among these people demonstrating to preserve the value of confederate monuments for science. It's also somewhat telling that I'm having trouble finding a single woman among that side of the protest.

Given such a protest–and even if you disagree with the above, please entertain this as a hypothetical–what would be the makeup of the group of people opposing such a protest? It seems to me that, in principle, everybody who disagrees with the far-right ideology of these protesters could, or even should, be among the counter-protesters. You can be a Nazi, or you can be against Nazis. But I'm having a hard time imagining someone being neutral: "I think the idea of sending the jews to the gas chamber has potential, but I will reserve judgement until I have studied it in more detail" just doesn't seem like a common opinion.

And that's why people are so outraged with the President's "there are always two sides" equivocation: one side wants genocide, the other wants "no genocide". Even if both sides had been similarly violent (which they were not: only one committed a terror attack killing someone), they aren't comparable. Because for these Nazis, the opposition is in the way of their fantasy of a whites-only country, whereas for these opponents, the step after keeping the upper hand against the Nazis is "going home".