Well, running with the nerd example, non-nerds never talk about being non-nerds while the ostracize the nerds. They can treat nerds as weird and other and never think of themselves as a particular thing other than "normal". They don't have to think about that because they dominate the dialog and become the unmarked case.
That's why, e.g., people on the autism spectrum recently came up with the term "neurotypical", so that they could try to create an equal footing in the discourse. In that construction, aspies and neurotypicals are just different kinds of people, both equally "normal".
Casting what I'm saying as "White Man's Burden" is either incredibly ignorant or a dick move, and I can't tell which. But to be clear, I'm up to the opposite.
The whole "White Man's Burden" [1] shtick rests upon the notion that white people are better, and therefore should help the benighted dark peoples. I, on the other hand, think races are fundamentally equal, and it's only an accident of history that leaves me with more privilege. I'm thus using it here to educate white people ignorant of the US's long history of ingrained racial prejudice.
Non-white people need no education on the topic. In the same way that nerds are forced to understand what nerdiness means to non-nerds, non-white people are forced to understand white notions of race, even as most white people see themselves as typical, average, default, "normal" and so never think about the topic in any depth.
[1] https://en.wikipedia.org/wiki/The_White_Man's_Burden