You could say the same for any pejorative, imagine if xyz was a treated as a slur, we’d have lost the forest for the trees.

And yes, if you said “don’t be a cripple” to your friend and a guy physical disability overheard it, how do you think that would make them feel? Why would you want to arbitrarily hurt someone, twisting a verbal knife, just to razz your friend.