This assumes a whole lot. How do you know for certain that the person is treating you differently from others because of your background? Depending on the person and the prevailing culture it may be perfectly normal to more or less ignore someone outside the office. If you're constantly on the lookout for some nefarious activity you may start to interpret it where it doesn't actually exist.

I've definitely done the same and have had the same done to me even with people who I am rather friendly with in the office (across all identity characteristics). I'm more introverted than most and generally shy away from a lot of social interactions unless I have to or would obviously come across as impolite. The prevailing culture in my location is to generally leave people alone outside the office as engaging in smalltalk can come across as being annoying when they probably just want to get where they're going, indulge in their phone, or are mentally preoccupied with what they should do for dinner that evening.