I think a lot of it has to do with the environment where you grew up and/or have spent a lot of time. I'm white and grew up in a heavily Asian community; when I'm the only white person in a mostly Asian group, I don't feel self-conscious, but when I'm in a mostly black or Latino group, I usually do.