One solution to this issue is to use an emoji font that is designed to be color-free, like one of the variants of OpenEmoji.