That’s why society as a whole makes that decision about what they consider correct and what they consider incorrect through laws and ethical standards. Unless you are in a dictatorship, laws and ethical standards are representative of the majority.
If an individual can’t decide for another, no assemblage of individuals can legitimately claim that authority either.
I certainly don’t want to live in a place where it takes hundreds of years for the majority to finally decide that, for example, racial minorities really do deserve equal protection under the law (or a hundred years to reach the point where it is not legal to enslave them).
Surely you see the problem.
Ultimately power has to rest somewhere. If it's nowhere then someone will come and take it.
The modern alternative isn't some panacea. It's the corporation and the indifference of rights so long as it doesn't negatively impact the bottom line.