Socially, we have countries with millions of people living under one flag.
It turns out the hypergamy negatively affects the average man more than the elite man. So how do you build a system the ensures that the vast majority of men don't die off? You must control for hypergamy.
Polygamy is a smaller issue, because the number of men who act in a manner that breaks a society is naturally much smaller than the number of women that can act in a manner to break society.
Take 1000 people. 500 of them are men. 50 of them can show interest to the vast majority of the women, and the women will gladly oblige. So here, we have 50 people (the elite men) that can cause unstable families.
Of the 500 women, each of them is trying to pick the best man given her qualities. So here, we have 500 people that can cause unstable families (all the women).
This would ultimately cause a large society with no control over hypergamy to dwindle into a small population over time. If we believe for a second people are pairing off "because they have matching personalities", we're all going to have a really bad time.