Yes, that is literally the concern. Activists hurl around the Nazi label with reckless abandon, showing little to no restraint when foisting the label on people they disagree with. There is also a sentiment taking hold that not being enthusiastically anti-Nazi makes you a Nazi sympathizer.
So yes, people have some reasonable reservations about where this could go, without having any real Nazi sympathies.
”wow, if they could do that to heretics, they could do the same thing to me (not a heretic)!”