Yeah that's definitely an understandable feeling. But on the other hand, if representation matters (as others in this thread claim), it seems hard to please everyone? Supposing you had a group that's 100% white by accident, and they know representation matters. Because they're 100% white, they're starting from a disadvantage: If you want representation, it helps to have representation already! So from this perspective, being 10% nicer to non-whites who join the group seems like a natural solution, to overcome the chicken-and-egg problem.
Maybe the right compromise is to just be especially careful not to be rude to people who are underrepresented in the group, and not worry about it beyond that?