"Hate speech" isn't just hateful speech, it's a specific term with a specific meaning. Being a nazi isn't an inherent characteristic of a person, it's an affiliation or ideology that they consciously choose.
Sure.
But the overwhelming majority of people called "nazis" by their political opponents have objectively not chosen anything remotely of the sort.