The only people saying that are members of right wing extremist organizations that have, as their guiding principle, the genocide of non-white people. These are groups that planned and trained to wage war against counter-protestors, and for weeks leading up to the event posted to their social media about their weapons, smashing people's heads, etc.
I've read a number of reports of the events in Charlottesville, and the overwhelming feeling I came away with was that antifa saved lives, in this case. One can reasonably criticize other times antifa has shown up; Berkeley, for example. But, I'm on team antifa in Charlottesville. I sure as hell don't want to be on the side of the Nazis who murdered someone.
More eyewitness testimony about antifa in Charlottesville:
http://www.slate.com/articles/news_and_politics/politics/201...
Edit: I think it's really important to recognize that violence is what Neo-Nazis are about. They are not rallying to express opinions, they are rallying to recruit and build a network of impressionable young men who will commit violent acts in pursuit of their goals. They were there specifically to find more people like James Alex Fields. To frame it as a peaceful political rally gone bad is to wholly misunderstand the threat Neo-Nazi organizations pose to our democracy.