A significant number of Americans and Europe whose ancestors weren't directly persecuted by nazis, but who fought in WW2 against the nazis and fascists, and they also find the swastika offensive.

Also note that the nazis also murdered Romani, Sinti, Poles, Slavs in concentration camps and ghettos. They also murdered and starved the general civillian populations of areas they took over.