The people working in sweatshops start as young children, often sold by their parents to pay debts. Often pay is withheld and they just become slaves or they are fed just enough to keep their fingers moving and their brains starved. The sweatshop may provide enough income for that child to grow up to produce more children, who then become sweatshop workers themselves, child prostitutes, or slaves, or they may just die on the job.
Nutrition deficiencies and stress get baked into their genetics: congratulations you now have a permanent underclass living worse than some animals, and similar to livestock, crowded populations, poor medical administration, and hellish living conditions result in festering and evolving pathogens that you and Paul Krugman may soon be more interested in. The result of providing just enough for them to survive to make children is not "less poverty", it's more poverty, more death, more child slavery or prostitution, and if emotional arguments don't register with you, exotic pandemics coupled with antibiotic misuse just might.
Secondly, America does not have a clothes shortage. The unfathomable amount of (often expensive) clothing thrown away by both consumers and suppliers in disgusting. Shrewd families from the lower income to the middle class buy used or keep hand-me-downs. Once or twice during move out season on campus my partner and I raid the rich dormitories and find upwards of thousands of dollars worth of designer clothing among other things. Many stores throw their out of season clothing straight into the dump.
The defense of a situation where hundreds of millions live in hell to produce mountains of goods that end in landfills, on an economic argument, is absurd.