You're looking at this backwards. It's a matter of protecting the vulnerable. But sure, let's pretend you're the one being disadvantaged by society.
If the point of a union is to protect me and my interest as a worker how does it do that by saying "you may be a higher performer but you're not the right <skin-color|religion|gender|sexual-orientation|etc>"?