I suppose one could use it derogatorily, but that's not my experience.
I assure you that context is everything. When Jews say "Jews", we just mean Jews. And when lots of goyim say "Jews", they mean "sub-human monsters".
Please feel free to censor your own language. And if you see me use "goyim" to be offensive, feel free to call me out on that.