I'm glad you've never heard "goyim" used in a derogatory manner, whereas I have. Tone is difficult to infer from plain text. To avoid confusion, I recommend you use other groups and races' preferred labels when speaking about them in public. This is advice I follow and have found to be a good rule.

Just so you know I'm not making this up; Merriam-Webster lists "goy" as "sometimes disparaging":

https://www.merriam-webster.com/dictionary/goy

Use of this word in 2017 got Huffington Post in trouble with the Anti-Defamation League, and others who felt it was over the line:

https://jewishnews.timesofisrael.com/huffpost-bannon-headlin...