Have you seen what the average poly person looks like? Caroline Ellison and Sam Bankman-Fried are the most famous poly people I can think of, and this is despite an obscene amount of wealth to make them look good.

So no, they're not taking partners from anybody. I don't see how society is harmed if the absolute bottom dregs shack up together instead of staying alone forever.

Whenever I've seen been subjected to photos of a group of polys, the men seem to have a similar look to the one the protagonist had in the closing shot of Old Boy (https://www.youtube.com/watch?v=l0nn-WgLOTY). Certainly their parents do.