Who gets hurt if social structure is affected by the apparent identity of masking? There are plenty of instances of catastrophic social division due to physical differences.
I think your choice should be your own, and based on health and other reasons. But it's pretty myopic to ignore the history of superficial social striations.