Let me try to address your point.
I agree that actually saying "How is it my problem" comes across antagonistic which wasn't my intent. Here is what I meant.
In the conversation about 'diversity' in tech today we really are talking about lack of black/hispanic individuals & women. So these are the three 'qualified' groups whose feeling of isolation at work is really discussed (and if a person fits two categories - double the points).
My point is that there are potentially many more (say infinitely more) individuals who may feel isolated and out of place. For example:
Old people
Immigrants
Lovers of opera
Extremely tall / short / fat people
etc
So anybody really who feels they don't fit it has the right to say 'There is lack of diversity with regards to X and I shouldn't change - we need to have more of X around so I feel comfortable'.
How would you respond if your coworker said "I'm a 55yo Scandinavian who loves opera and I feel like I don't fit in. Therefore I feel it's the responsibility of [company / society] to get more senior Scandinavian opera lovers in the field so I am comfortable"?
But somehow since the 3 groups in question are seen as disadvantaged we feel that the conversation is legitimate.