It's no less arbitrary than deciding that gypsies, who engage in petty crime and mistreat their children, forcing them to become beggars and thieves themselves, should not reproduce. If anything, Down kids probably bring more good to society than gypsy kids. And forced sterilization is arguably more ethical than forced abortion.
Same things goes for alcoholics and drug users.
Privately, most people agree with this position. The idea that eugenics is inherently bad is very unsound and doesn't withstand scrutiny.