You're right that it is used commonly, but we have the chance to be in a position to know better (thanks to our cross-cultural and larger perspective).

The right term is gaikokujin, but if we speak in English there is no reason to not just say "foreigner".

"gaijin" has the vibes of "outsider", and is never used in public speech other than by xenophobic speakers.

In private or familiar conversations it is commonly used, yes.