Having a physical advantage is not dominance. It is merely capability with regards to the objective in play, which is where the argument of division of labour comes into play.
You want to have people play to their capabilities and (metaphorical) strengths. It’s how a society (or any cohesive unit of people, actually) maximizes its effectiveness and utility for everyone concerned.
To not do so is akin to having two people paint a wall, and for the sake of appearances you have the tall one paint near the floor and the short person paint up near the ceiling. Unless other attributes significantly overcome the massive disadvantages of the height difference, it would be madness to force them to paint the wall like this.