My sense is the general argument goes: if you, as a member of the dominant race/class do something (adopt a style of dress, way of wearing your hair, style of music, etc.) which, when it is done by its originators leads to discrimination against them (for example people of African descent wearing their hair in dreads) but becomes cool and even profitable for you (think Elvis playing R&B), that's appropriation. I agree it's tricky because I don't think we want to advance essentialist arguments in general, but the focus is more on the unfairness than the "only X can do Y".