I've been in Japan for 10+ years and only speak my mother tongue of Finnish while talking on the phone with my parents. Very little loss, doesn't feel awkward, but I sometimes forget very rarely used words.

If I have a half an hour conversation, there might be one word that a normal person should know, but that I've forgotten. Comes as a bit of a shock to me when this happens.