You're observing a response to authoritarianism on the left. We can expect increased conflict between ideologues on both sides of the political spectrum.
The actual number of neo-Nazis is very small but we hear more about them because they're the greatest troublemakers and because of history.
There are many disgruntled conservatives and former lefists who are also upset about authoritarism on the left. The neo-Nazis are using their sentiment as fuel, so to speak.
We can see a similar thing happening with Antifa emboldend by the current clout of social Marxism.