I never get offended when I am asked. Why would I? It’s natural curiosity IMHO. And actually makes small talk easier as it’s a subject I know and can chat about :)
But I don’t know, if I told you I was from Greece, would my visa be the first think that would pop into your mind for small talk?