Yes, when you're Jewish and there's an armed mob of white supremacists on the street, the last thing we want to do is insult them by improper classification.
People did fly the Confederate flag though, so "white supremacists" seems beyond a reasonable doubt.
But if it would be significant that people were flying actual Nazi flags, then it must also be significant if they weren't. To some extent. Other demonstrations did have such flags.
This is an example of a more general pattern that's been bothering me recently. It seems like a contradiction to claim that something is a heinous offense, while simultaneously claiming it doesn't matter whether a given person or entity is really guilty or not. This is saying that guilt is and isn't important at the same time.