For some people, race, fighting for for recognition, let alone equality, is a daily battle. You may live and work far from this conflict, but it exists, and in some part the diversity modifiers for emoji provide folks with empowerment. Don't take that away over limitations in the spec.
There are billions now who own smart phones, and want that funny Japanese Telco encoding standard to reflect their world too.
This is not a technical or standards problem, and the fitzpatrick modifiers do not decrease functionality of unicode.
I feel my point is to chill, and consider how functionality outside your perceived value might bring others joy.