It's well known that in some modern US polygamy situations, there is a great deal of abuse of power, both in terms of controlling the wives, as well as controlling and abusing the young men who will not be allowed to have a wife. This further increases the societal costs and leads to more abuse of power, which is not what we need.
edit: I should add: It's a numbers game. Given that on average, there tends to be just slightly more women born than men, what happens with all the extra unmarried men (or women, though this is rarer)?