> I causally used the "r-word" and didn't even think anything of it

You casually called someone a "retard" (I assume that's the word) and someone took that wrongly? Of course they did! Even when "retard" was an acceptable put-down, it was still a put-down.

Maybe the way to avoid the "minefield" you speak of is to not put others down? Even deservedly! Just try to be nicer to everyone, even if they are not, themselves, nice.